Vehicle control and guidance

ABSTRACT

A vehicle computing system may identify a scenario in an environment that violates an operating constraint. The vehicle computing system may request remote guidance from a guidance system of a service computing device. The vehicle computing system may receive input from the guidance system including one or more waypoints and/or associated orientations for the vehicle to navigate through the scenario. The vehicle computing system may be configured to validate the input. A validation may include processing the input to determine whether the waypoint(s) and/or orientation(s) associated therewith may cause the vehicle to violate a safety protocol. Based on a determination that the input will not cause the vehicle to violate the safety protocol, the vehicle computing system may control the vehicle according to the input, such as by causing a drive system to operate the vehicle to each waypoint at the associated orientation.

BACKGROUND

Vehicles operate in dynamic environments in which conditions are oftenchanging. Among the changing conditions are road blockages due toconstruction, accidents, and the like. Autonomous vehicles may beprogrammed to react to the changing conditions, while maintaining thevehicle within a designated operations protocol. The operations protocolmay include defined rules of operation, such as not operating thevehicle into a lane designated for opposite direction traffic. However,oftentimes a detour around an accident, construction zone, or the likemay indicate a route outside the parameters of the operations protocol.Unauthorized to operate outside the bounds of the operations protocol,the autonomous vehicle may be unable to proceed around the obstacle andmay thus be stuck in place.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical components or features.

FIG. 1 is an illustration of an autonomous vehicle in an environment, inwhich a remote vehicle guidance system may be used to assist in guidingthe vehicle through an environment.

FIG. 2 is an illustration of an example service computing device with agraphical user interface in which an operator may input waypoint and/ororientation guidance for a vehicle operating in a remote guidance mode.

FIG. 3 depicts an example process for providing incremental guidance toa vehicle.

FIG. 4 is an illustration of an example flow diagram depicting anoperator workflow for providing remote guidance to a vehicle via agraphical user interface.

FIG. 5 is an illustration of an example flow diagram depicting vehicleactions responsive to remote guidance input.

FIG. 6 depicts an example process for providing remote guidance to avehicle computing device.

FIG. 7 depicts an example process for controlling a vehicle based onguidance input received from a service computing device.

FIG. 8 is a block diagram of an example system for implementing thetechniques described herein.

DETAILED DESCRIPTION

This disclosure is directed to techniques for remotely providingincremental guidance to a vehicle operating in an environment that hasencountered a scenario that is difficult to navigate (e.g., in which aplanner is unable to plan a route according to a set of drivingpolicies, or otherwise). The vehicle may include an autonomous orsemi-autonomous vehicle with a vehicle computing system configured torequest guidance from a remote vehicle guidance system based on anobstacle encountered in an operating environment. The remote vehicleguidance system may include a graphical user interface (GUI) throughwhich an operator may input suggested waypoints and/or orientations(e.g., vehicle yaw at a corresponding waypoint) for the vehicle tonavigate through the scenario (e.g., around one or more obstacles). Thevehicle may receive the suggested waypoints and/or orientations andverify that navigating along the path defined by the suggested waypointsand/or orientations satisfies a safety protocol of the vehicle.Accordingly, the techniques described herein improve the safety of thevehicle operating in the environment.

The vehicle computing system may receive sensor data from one or moresensors (e.g., cameras, motion detectors, lidar, radar, time of flight,etc.) of the vehicle. The vehicle computing system may determine, basedon the sensor data, that the vehicle is approaching an obstacle on theroute around which the vehicle may not be configured to determine aviable path (e.g., road blockage, construction zone in the road,driveline beyond vehicle line-of-sight (e.g., sensor vision impaired dueto elevation of road and/or turns in a road), etc.), referred to hereinas a remote guidance scenario. The remote guidance scenario may includea scenario in which the vehicle computing system is not programmed toperform (e.g., unknown scenario) and/or a scenario that appears toviolate an operations protocol (e.g., a policy). The operations protocolmay include one or more rules governing actions the vehicle may or maynot take such as not crossing a double yellow line, not driving intooncoming traffic lanes, not exceeding the bounds of a drivable surfaceof a road, or the like. For example, the vehicle computing system maydetect that a road on which the vehicle is traveling is blocked and thata way around the blockage would include driving into lanes designatedfor traffic traveling in an opposite direction, violating an operationsprotocol.

In some examples, responsive to detecting the remote guidance scenario,the vehicle computing system may automatically connect to a servicecomputing device configured with the GUI. In various examples, a vehicleoperator and/or safety observer may detect the remote guidance scenarioand may connect to the service computing device. In some examples, thevehicle computing system may send a request for guidance to the servicecomputing device. In various examples, prior to receiving a response(e.g., guidance input) from the service computing device, the vehiclecomputing system may determine a solution for navigating the remoteguidance scenario. In such examples, the vehicle computing system maycontrol the vehicle according to the solution. In some examples, thevehicle computing system may send a second message to the servicecomputing device indicating that guidance input is no longer needed(e.g., indicating that the vehicle is proceeding without remoteguidance). In some examples, the vehicle computing system may proceed tocontrol the vehicle without sending the second message to the servicecomputing device. In such examples, at least one of the vehiclecomputing system or the service computing device may determine that theguidance input is not necessary and may disconnect from the othercomputing device.

In some examples, responsive to receiving the request for guidance, theservice computing device may determine that one or more conditions forproviding guidance are met. The condition(s) may include a connectionthreshold (e.g., threshold latency, threshold bandwidth, etc.), avehicle velocity below a threshold velocity (e.g., 15 miles per hour, 20kilometers per hour, etc.), a threshold number/type of sensoroperational (e.g., threshold of 4 cameras operational, all vehicle lidarsensor operational, etc.), a lack of vehicle health related faults(e.g., faults related to inconsistent sensor readings, one or moreinoperative sensors, fault associated with a collision avoidance system,planner system, and/or other vehicle system, etc.), and/or otherconditions that may impact the effectiveness of the remote guidancesystem. In some examples, responsive to a determination that a conditionis not met, the service computing device may not connect with thevehicle computing system. In some examples, responsive to adetermination that a condition is not met, the service computing devicedeny the operator access to the GUI and/or deny input into the GUI. Insuch examples, denying access may enhance safety of vehicle operation,such as, for example, not permitting remote guidance without sufficientlatency to ensure that the vehicle computing system is able to respondin an adequate time frame to avoid unsafe outcomes. In some examples,responsive to a determination that the conditions are met, the servicecomputing device may launch the GUI.

In some examples, the GUI may include one or more windows depictingsensor data from one or more sensors on the vehicle. In at least oneexample, the GUI may include windows depicting streaming images capturedby a camera on the vehicle. The window(s) may depict data to assist theoperator in determining a path for the vehicle to take. For example, theGUI may include four windows depicting a left front view, front view,right front view, and a rear view associated with a vehicle. Theoperator may assess each of the views to determine a safe path for thevehicle to transit though the remote guidance scenario.

In various examples, the GUI may include a depiction of a top view ofthe vehicle and the remote guidance scenario in the environment. In suchexamples, the top view may include a computer-generated view of theenvironment based on sensor data. In some examples, the top view mayinclude a position of the vehicle with respect to one or more obstaclesof the remote guidance scenario. For example, a top view may depict avehicle approaching multiple orange cones blocking a pre-determined(e.g., planned) path of the vehicle. The top view may include thevehicle and the cones.

In various examples, the GUI may be configured to receive input from theoperator via the top view. In such examples, the operator may be able toprovide guidance to the vehicle to navigate through the remote guidancescenario, such as around obstacle(s) associated therewith. The input mayinclude one or more waypoints. The waypoint(s) may include locationsover which the vehicle will travel. In various examples, the waypoint(s)may each include stopping points. In such examples, the vehicle may beconfigured to stop at each of the waypoint(s), such as absent receipt orverification of a subsequent waypoint. In at least some examples, wherea subsequent waypoint is provided, the vehicle may continue planning andmoving between waypoints without stopping.

In various examples, the input may include an orientation (e.g., yaw) ofthe vehicle at an associated waypoint. In such examples, the operatormay be able to cause the vehicle to face a particular direction at theassociated waypoint. In some examples, the orientation may be determinedbased on a previous and/or subsequent waypoint. In such examples, theorientation at a particular waypoint may enable the vehicle to smoothlytransition between waypoints on a guidance route. In various examples,an orientation associated with a waypoint may be based on a preferredorientation should the vehicle be stopped at the waypoint. For example,a preferred orientation associated with vehicles located at leastpartially in a lane associated with opposite direction traffic may beparallel to the lane such that an amount of the vehicle located in thelane may be minimized.

In some examples, the operator may cause the service computing device tosend the waypoint(s) and/or orientation(s) to the vehicle computingsystem. In some examples, the service computing device may automaticallysend the waypoint and/or associated orientation responsive to input bythe operator. In such examples, based on a determination that thewaypoint is confirmed in a particular place, and/or the orientation isconfirmed, the service computing device may automatically send thewaypoint and/or orientation. In some examples, the waypoint and/orassociated orientation may be sent responsive to receiving a sendcommand (e.g., selection of a send option) from the operator. In suchexamples, the operator may be able to send waypoints individually and/orin groups.

In some examples, the service computing device may be configured tovalidate the waypoint(s) and/or orientation(s) prior to sending theguidance data to the vehicle computing system. In some examples, thevalidation may include an initial verification that the waypoint(s)and/or orientation(s) pass a safety protocol associated with thevehicle. The safety protocol may include a pre-determined set ofconditions for ensuring vehicle safety. The safety protocol may includeone or more threshold distances for the vehicle to maintain away fromdynamic and/or static objects, maximum yaw rates and/or other criteriaassociated with safe navigation of the vehicle through the remoteguidance scenario. In various examples, the vehicle computing system mayreceive the waypoint(s) and/or orientation(s) from the service computingdevice and may validate the waypoint(s) and/or orientation(s). In suchexamples, the vehicle computing system may verify that the waypoint(s)and/or orientation(s) pass the safety protocol and may safely beperformed by the vehicle.

In some examples, the vehicle computing system may be configured toverify that the waypoint(s) and/or orientation(s) based at least in parton a remote guidance protocol. In such examples, the vehicle computingsystem may validate the waypoint(s) and/or orientation(s) based in parton a determination that the waypoint(s) and/or orientation(s) pass theremote guidance protocol. The remote guidance protocol may include oneor more limitations on vehicle movement while operating in the remoteguidance mode. The remote guidance protocol may include a path deviation(e.g., distance from a path between two waypoints the vehicle maydeviate, such as to circumvent obstacles, ½ width of the vehicle, 1vehicle width, etc.), response to static object detection (e.g., attemptto circumvent static objects to get to a follow-on waypoint, response todynamic object detection (e.g., yield to agents predicted to enter apath corresponding to the waypoints), follow dynamic objects moving inthe same direction as the vehicle (e.g., in the path corresponding tothe waypoints), yield as appropriate for right of way at intersections(e.g., four way stops, traffic lights, roundabouts, etc.), stop forwardmotion if no further guidance received (e.g., another waypoint,instruction to resume navigation, etc.), or the like.

Additionally or in the alternative, the vehicle computing system may beconfigured to validate the waypoint(s) and/or orientation(s) based atleast in part on a kinematics validation (e.g., dynamics validation,kinematics/dynamics checks). The kinematics validation may includedetermining that the vehicle is capable of performing a transition froma first position (e.g., an initial position, position associated with awaypoint, etc.) to a second position (e.g., first waypoint, subsequentwaypoint, etc.). In some examples, the capability of performing thetransition may include determining whether the vehicle computing systemcan generate one or more trajectories for the vehicle to perform toarrive at the waypoint(s) in the orientation(s) based on physicalconstraints. The physical constraints may include a maximum yaw rate,maximum lateral distance, maximum turn angle, and/or other constraintsassociated with vehicle maneuvering.

Responsive to a determination that a waypoint and/or associatedorientation violates the safety protocol and/or the remote guidanceprotocol, the vehicle computing system may reject the waypoint. In someexamples, the vehicle computing system may not proceed to the rejectedwaypoint, such as by not proceeding from a stopping point (e.g., a stopline, stop sign, traffic light, etc.) or stopping at a waypoint prior tothe rejected waypoint. In some examples, the vehicle computing systemmay send a message to the service computing device indicating therejection of the waypoint. In some examples, the message may include areason for the rejection. For example, the vehicle computing system maydetermine that a waypoint is less than a threshold distance from acement barrier. Based on the determination, the vehicle computing systemmay reject the waypoint and send a message to the service computingdevice indicating that the distance is less than the threshold distance.

Responsive to a determination that the waypoint(s) and/or associatedorientation(s) pass the safety protocol and/or the remote guidanceprotocol, the vehicle computing system may accept the waypoint(s) and/ororientation(s). The vehicle computing system may process the waypoint(s)and/or orientation(s) and generate one or more trajectories for thevehicle to transit between waypoints. For example, the vehicle computingsystem may receive a first waypoint and a first orientation associatedwith the first waypoint. The vehicle computing system may determine afirst trajectory (or set of trajectories) to transit from an initialposition to the first waypoint at the first orientation.

Based on the generated trajectories, the vehicle computing system maycontrol the vehicle to the waypoint(s) at the associated orientation(s),such as via one or more drive systems. In various examples, the vehiclecomputing system may control the vehicle at a pre-determined velocity(e.g., 20 kilometers per hour, 15 miles per hour, 10 miles per hour,etc.). In some examples, the pre-determined velocity may include amaximum velocity associated with remote guidance. In various examples,the vehicle computing system may control the vehicle at a velocityassociated with the environment and/or the scenario. In at least oneexample, the vehicle computing system may control the vehicle at thelesser of the pre-determined velocity or the velocity associated withthe environment and/or the scenario. For example, a vehicle computingsystem, when operating in a remote guidance mode, may operate at apre-determined maximum velocity of 15 miles per hour. The vehiclecomputing system may detect a remote guidance scenario and requestguidance input from a service computing device. The vehicle computingsystem, and/or the operator may determine, based on sensor data, thatthe remote guidance scenario has associated therewith a speed limit of 5miles per hour. Based on the speed limit of 5 miles per hour being lessthan the pre-determined maximum velocity of 15 miles per hour, thevehicle computing system may limit the velocity of the vehicle to 5miles per hour while navigating through the remote guidance scenario.

In various examples, the vehicle computing system may be configured tocontinuously and/or periodically verify that the vehicle will notviolate the safety protocol and/or remote guidance protocol whilenavigating through the remote guidance scenario (e.g., travelingwaypoint to waypoint). For example, a vehicle computing system maycontrol a vehicle from a first waypoint to a second waypoint. Thevehicle computing system may detect a pedestrian with a trajectory thatappears to intersect with a trajectory associated with the vehicle. Thevehicle computing system may determine an action to take based on thepedestrian, such as yielding to the pedestrian.

In various examples, the service computing device may determine that ascenario is complete (e.g., vehicle passes obstacle, can resume normaloperations) and may release the vehicle from the remote guidance. Insome examples, scenario completion may be based in part on a generatedwaypoint being within a threshold distance (e.g., 0.5 meters, 0.5 feet,10 inches, etc.) of an original route associated with the vehicle. Insome examples, scenario completion may be based on sensor data input. Insuch examples, the operator may view the sensor data via the GUI todetermine that the scenario is complete. For example, the obstacle mayinclude a construction zone. A camera feed from the vehicle may depictan end of the construction zone behind the aircraft. Based on adetermination that the construction zone is behind the vehicle, theoperator may determine that the scenario is complete. In some examples,determination may be based on a planner system determining a trajectorywhich is allowable (e.g., doesn't violate a policy) and/or can continuealong a determined path.

In various examples, responsive to a determination that the scenario iscomplete, the service computing device may send a release signal to thevehicle. In such examples, responsive to the release signal, the vehiclecomputing system may be configured to resume navigation of the vehicle(e.g., autonomous control). In various examples, the service computingdevice may determine that the scenario blocked an initial route of thevehicle. In such examples, the service computing device may beconfigured to generate an updated route for the vehicle to arrive at apre-determined destination. In some examples, the service computingdevice may send the updated route to the vehicle computing system priorto sending the release signal. In some examples, the service computingdevice may send the updated route substantially concurrently with therelease signal (e.g., within 1 second, 2 seconds, etc.). Responsive toreceipt of the updated route, the vehicle computing system may controlthe vehicle along the updated route to the pre-determined destination.In some examples, the vehicle computing system may determine that thescenario blocked the initial route. In such examples, the vehiclecomputing system may be configured to generate the updated route tonavigate to the destination.

The techniques discussed herein may improve a functioning of a vehiclecomputing system. Traditionally, in control planning for an autonomousvehicle, a vehicle computing system may continually process sensor datato determine a viable route for the vehicle until it determines asolution. In examples in which the vehicle encounters a remote guidancescenario, the solution may be unattainable based on constraints put onthe vehicle computing system, such as in operations protocols. In suchexamples, the vehicle computing system may utilize a tremendous amountof computing resources without reaching a solution. Thus, the vehiclemay be stuck in a particular position on a planned route while thevehicle computing system continues to use processing power and/or memoryto identify a path through the remote guidance scenario.

The techniques described herein provide a technical solution to thetechnical problem encountered by the vehicle computing system due to theconstraints by utilizing remote computing resources to identify anear-real time solution for navigating through the remote guidancescenario. The graphical user interface on the service computing device,as described herein may process sensor data from the vehicle computingsystem and provide guidance to the vehicle computing system, therebyreducing the amount of processing power and/or memory used by thevehicle computing system, which in some examples, could otherwise bestuck in an infinite loop of processing without identifying a solution.

Additionally, the techniques described herein may increase the safety ofoperation of the autonomous vehicle through an environment. For example,the vehicle computing system may connect to the service computing devicefor guidance input based on a determination that further movement mayviolate an operations protocol. To ensure a maximum degree of safety,the vehicle computing system may request guidance from a remote operatorthat may be configured to evaluate a situation and determine a saferoute forward. Further, the vehicle computing system may receive theinput and may verify that the guidance input results in a safe operationof the vehicle prior to controlling the vehicle according to the input.

The techniques described herein may be implemented in a number of ways.Example implementations are provided below with reference to thefollowing figures. Although discussed in the context of an autonomousvehicle, the methods, apparatuses, and systems described herein may beapplied to a variety of systems (e.g., a sensor system or a roboticplatform), and are not limited to autonomous vehicles. In anotherexample, the techniques may be utilized in an aviation or nauticalcontext, or in any system using machine vision (e.g., in a system usingimage data).

FIG. 1 is an illustration of an autonomous vehicle 102 (vehicle 102) inan environment 100, in which a remote guidance system may determine adeviation from a planned route 104 (e.g., initial route, planned path,etc.) based on a detection of a remote guidance scenario 106 (e.g., anobstruction on the planned route 104 around which the vehicle may not beconfigured to determine a viable path). The remote guidance scenario 106may include a scenario in which a vehicle computing system is notprogrammed to perform and/or a scenario that appears to violate anoperations protocol (e.g., a trajectory required to navigate thescenario would violate the operations protocol (e.g., control policy)).The operations protocol may include one or more rules governing actionsthe vehicle may or may not take such as not crossing a double yellowline, not driving into oncoming traffic lanes, not exceeding the boundsof a drivable surface of a road, or the like. In some examples, theremote guidance scenario 106 may include a scenario in which an obstacleextends beyond a perception distance associated with the vehicle 102.For example, as illustrated in FIG. 1, the vehicle 102 may be unable toperceive the entirety of the remote guidance scenario 106 from aninitial position 108 (e.g., initial location and/or orientation).Additionally illustrated in FIG. 1, the remote guidance scenario 106 mayinclude one or more obstacles blocking the planned route 104 of thevehicle 102, such that navigating around the remote guidance scenario106 would result in the vehicle 102 operating in a lane designated foropposite direction traffic. As non-limiting examples, the remoteguidance scenario 106 may comprise cones, barricades, etc. whichdemarcate an altered lane boundary, workers signaling a direction forthe vehicle 102 to proceed, or otherwise a scenario which differs from aset of nominal driving scenarios.

In various examples, the vehicle computing system may detect the remoteguidance scenario 106 based on sensor data from one or more sensors. Thesensor(s) may include cameras, motion detectors, lidar, radar, time offlight, and/or other sensors mounted on the vehicle 102. As will bediscussed in greater detail below with respect to FIG. 8, a plannercomponent of the vehicle computing system may be configured to navigatethe vehicle 102 along the planned route 104. The planner component mayreceive processed sensor data from a perception component of the vehiclecomputing system and may determine one or more trajectories for thevehicle to travel along the planned route 104 based on the sensor data.The trajectories may include a direction, orientation, and/or speed thevehicle may travel through the environment.

The trajectories associated with the vehicle operation may be determinedbased in part on an operations protocol (e.g., one or more operatingconstraints) placed on the vehicle computing system. The operationsprotocol may include one or more rules governing actions the vehicle mayor may not take such as not crossing a double yellow line, not drivinginto oncoming traffic lanes, not exceeding the bounds of a drivablesurface of a road, not exceeding a speed limit, or the like. Forexample, an operations protocol may include a limitation for the vehicleto remain on a paved surface of a road. The vehicle computing system maythus be configured to generate one or more trajectories that maintainthe vehicle 102 on the paved surface of the road.

In various examples, the vehicle computing system may receive the sensordata and determine that planned route 104 is blocked, such as by anobstacle, or otherwise has a high uncertainty for proceeding, cannotproceed without violating a control policy, or the like. In someexamples, the vehicle computing system may process the sensor data andattempt to generate one or more trajectories (that satisfy operationsprotocol) for the vehicle to navigate around the obstacle. However, insome examples, the scenario encountered, such as the remote guidancescenario 106 depicted in FIG. 1, may not include a possible trajectory(or path) that satisfies the operations protocol. In such examples, thevehicle computing system may identify the scenario as the remoteguidance scenario 106 and may cause the vehicle to come to a stop atinitial position 108. In various examples, based on a detection (e.g.,identification) of the remote guidance scenario 106, the vehiclecomputing system may cause one or more hazard signals (e.g., hazardlights, audio signal, etc.) to emit from the vehicle.

Based on detection of the remote guidance scenario 106, the vehiclecomputing system may automatically connect to the service computingdevice, such as via the one or more networks. The network(s) may includepublic networks such as the internet, private networks such as aninstitutional and/or personal network or some combination of public andprivate networks. The network(s) may also include any type of wiredand/or wireless network, including but not limited to satellitenetworks, cable networks, Wi-Fi networks, WiMax networks, mobilecommunications networks (e.g., 3G, 4G, 5G, etc.), local area networks(LAN), wide area networks (WAN), or any combination thereof. In someexamples, the vehicle computing system may connect to the servicecomputing device based on input received from an input/output device,such as a by a press of a button. For example, a safety observer,vehicle operator, or passenger in the vehicle may press a button toconnect to the service computing device, such as for remote guidance.

In various examples, the vehicle computing system may send signalindicating the remote guidance scenario 106. In some examples, thesignal may include a request for guidance data (e.g., guidance input)from the service computing device. In some examples, responsive toconnecting with and/or receiving a request for guidance data from thevehicle computing device, the service computing device may verify thatconditions for guidance are met. The conditions for guidance may includesystem checks to ensure that the service computing device caneffectively provide guidance to the vehicle 102. The conditions mayinclude a vehicle velocity below a threshold (e.g., less than 15 milesper hour, less than 20 kilometers per hour, etc.), a threshold numberand/or type of sensors are available (e.g., minimum of four camerasproviding forward, aft, left, and right views operational, sensorsassociated with aerial perspective view operational, etc.), thresholdbandwidth (e.g., equal to or greater than 10 megabytes per second, 15megabytes per second, etc.), threshold latency (e.g., equal to or lessthan 300 milliseconds, 250 milliseconds, etc.), no vehicle healthrelated faults (e.g., steering column fault, etc.), or the like.

In some examples, responsive to a determination that one or moreconditions are not met, the service computing device may deny aconnection (e.g., not connect) with the vehicle computing system. Invarious examples, responsive to a determination that one or moreconditions are not met, the service computing device may deny anoperator access to and/or deny input (e.g., disable input capability)into the graphical user interface (GUI) of the service computing device.In some examples, the service computing device may send the vehiclecomputing device an indication that the condition(s) are not met and/orthat the remote guidance system is unavailable.

In various examples, responsive to a determination that the conditionsare met, the service computing device may launch the GUI on a display.In various examples, the GUI may include one or more windows depictingsensor data from the sensor(s) of the vehicle 102. For example, thewindow(s) may each depict real-time or near real-time (e.g., such asbased on latency) camera images captured by cameras mounted on thevehicle 102. In various examples, the data depicted in the window(s) mayprovide the operator with information to assist in determining a patharound the remote guidance scenario 106.

In various examples, the GUI may include a depiction of a top-down viewof the vehicle 102 (or otherwise configurable by the remote operator)and the remote guidance scenario 106 in the environment 100. In suchexamples, the top view may include a computer-generated view of theenvironment based on sensor data. In some examples, the top view mayinclude a position of the vehicle 102, such as initial position 108,with respect to one or more obstacles of the remote guidance scenario106. As will be discussed in further detail with regard to FIG. 2, theGUI may be configured to receive input from the operator via the topview. In such examples, the operator may be able to provide guidance tothe vehicle to navigate through the remote guidance scenario 106, suchas through a drive corridor 110. The drive corridor 110 may represent adrivable surface around the remote guidance scenario 106 in which theoperator may input one or waypoints 112. The waypoint(s) 112 may includelocations (e.g., X, Y, Z coordinates, etc.) over which the vehicle willtravel. In various examples, the waypoint(s) 112, such as waypoint112(1) through waypoint 112(N), may each include stopping points. Insuch examples, the vehicle may be configured to stop at each of thewaypoint(s) 112, such as absent receipt or verification of a subsequentwaypoint 112. For example, the operator may input waypoint 112(1) andabsent receipt of waypoint 112(2), the vehicle computing device maycause the vehicle 102 to stop at waypoint 112(1) until further guidanceis received.

In various examples, the input may include an orientation 114 (e.g.,yaw) of the vehicle 102 at an associated waypoint 112. In such examples,the operator may be able to cause the vehicle to face a particulardirection at the associated waypoint 112. In various examples, anorientation 114 associated with a waypoint 112 may be determined basedon a preferred orientation should the vehicle be stopped at thewaypoint. For example, to minimize exposure to oncoming traffic whennavigating around a remote guidance scenario 106, a preferredorientation associated with a vehicle 102 located at least partially ina lane associated with opposite direction traffic may be substantiallyparallel (e.g., within 5 degrees, etc.) to the lane such that an amountof the vehicle 102 located in the lane may be minimized. Accordingly, atleast to minimize exposure to opposite direction traffic, orientation114(2) associated with waypoint 112(2) may be substantially parallel tothe lane. In at least some examples, such orientations may be providedto enable the vehicle to find a trajectory which connects subsequentwaypoints.

In some examples, the orientation 114 may be determined based on aprevious and/or subsequent waypoint 114. In such examples, theorientation 114 at a particular waypoint 112 may enable the vehicle tosmoothly transition between waypoints on a guidance route. For example,a first orientation 114(1) associated with a first waypoint 112(1) mayrepresent a straight line or spline in the direction of a secondwaypoint 112(2). A second orientation 114(2) associated with the secondwaypoint 112(2) may represent a spline in the direction of the thirdwaypoint 112(3), and so on. As will be discussed in greater detailbelow, the vehicle computing device may receive the waypoint 112 and/orthe orientation 114 and may generate one or more trajectories tosmoothly control the vehicle 102 between waypoints and/or orientations.

In various examples, the operator may place a waypoint 112, such aswaypoint 112(1) at a position in the drive corridor 110. The drivecorridor 110 may include a determined section of the road that thevehicle 102 may transit through to navigate the remote guidance scenario106. In various examples, the service computing device may generate thedrive corridor 110 based on sensor data received from the sensors on thevehicle. In some examples, the service computing device may generate thedrive corridor 110 based on one or more detected obstacles in theenvironment 100. For example, a remote guidance scenario 106 may includeobstacles on a right side and a left side of the vehicle 102. The drivecorridor 110 and/or a width thereof may be defined by the distancebetween obstacles and/or a safety buffer (e.g., pre-determined distanceto maintain the vehicle 102 away from the obstacles). In variousexamples, a width of the drive corridor 110 may be determined based on awidth of the vehicle. In some examples, the width of the drive corridor110 may additionally include a safety buffer on either side of thevehicle 102. In various examples, the drive corridor 110 may representan area in which an operator may input waypoints 112. In such examples,the GUI may not permit waypoints 112 to be input outside of the drivecorridor 110.

In various examples, the GUI may limit a distance D (e.g., 10 feet, 15feet, 7 meters, etc.) between waypoints 112 and/or between an initialposition 108 and a first waypoint 112(1). In some examples, the GUI maynot permit waypoints 112 to be input more than the distance D away fromone another. In some examples, the GUI may activate an area with aradius of D around the vehicle 102 in which the operator may input awaypoint 112. In some examples, the GUI may provide a notification tothe operator, such as in a pop-up notification, that an input is outsidethe distance D and will not be accepted. In various examples, limitingthe distance between waypoints may enable the incremental guidance ofthe vehicle 102 through the environment 100 and may ensure that thevehicle 102 may safely maneuver the remote guidance scenario 106.

In various examples, the operator may input a waypoint 112 and/or anassociated orientation 114, such as waypoint 112(1) and orientation114(1), into the GUI. As will be discussed in greater detail below withregard to FIG. 2, the operator may input the waypoint 112(1) byselecting a location in the environment 100 for the waypoint 112(1) andconfirming the location. In some examples, confirmation of the locationmay include a click of the mouse at the location and/or a selection of aconfirmation option on the GUI. In various examples, responsive toreceiving confirmation of the waypoint 112 and/or orientation 114, theservice computing device may validate the waypoint 112 and/ororientation 114. A validation of a waypoint 112 and/or orientation 114may include a verification that the waypoint 112 and/or orientation 114satisfies a safety protocol of the vehicle. The safety protocol mayinclude one or more rules associated with the safe operation of thevehicle 102, such as a location of the waypoint 112 on a road (e.g., notless than ½ width of the vehicle 102 from the edge of the road ordrivable surface), waypoints 112 within a threshold distance of aninitial position 108 or within a threshold distance of one another, oneor more threshold distances for the vehicle 102 to maintain away fromdynamic and/or static objects, maximum yaw rates, and/or other criteriaassociated with safe navigation of the vehicle 102 through theenvironment 100.

In some examples, the operator may cause the service computing device tosend the waypoint(s) 112 and/or orientation(s) 114 to the vehiclecomputing device. In some examples, the service computing device mayautomatically send the waypoint 112 and/or associated orientation 114responsive to input by the operator. In such examples, based on adetermination that the waypoint 112 is confirmed in a particular place,and/or the orientation 114 is confirmed, the service computing devicemay automatically send the waypoint 112 and/or orientation 114. Invarious examples, one or more waypoints 112 and/or associatedorientations 114 may be sent based on a determination that the point arevalidated (e.g., pass a safety protocol, remote guidance protocol,and/or a kinematics validation). In some examples, the waypoint 112and/or associated orientation 114 may be sent responsive to receiving asend command (e.g., selection of a send option), such as that input bythe operator. In some examples, the GUI may be configured to sendwaypoints 112 and/or associated orientations 114 individually and/or ingroups.

In various examples, the vehicle computing system may receive thewaypoint(s) 112 and/or orientation(s) 114 from the service computingdevice and may validate the waypoint(s) 112 and/or orientation(s) 114.In such examples, the vehicle computing system may verify that thewaypoint(s) 112 and/or orientation(s) 114 pass the safety protocol andmay safely be performed by the vehicle 102 and/or a kinematicsvalidation (e.g., that the vehicle 102 can achieve the desired waypointbased on physical constraints). Responsive to a determination that awaypoint and/or associated orientation violates the safety protocol, thevehicle computing system may reject the waypoint. In some examples, thevehicle computing system may not proceed to a rejected waypoint, such asby not proceeding from a stopping point (e.g., initial location 108) orstopping at a waypoint prior to the rejected waypoint. For example, thevehicle computing device may receive a first waypoint 112(1) and a firstorientation 114(1) and may determine that the first waypoint 112(1) andthe first orientation 114(1) are valid (e.g., pass safety protocol). Thevehicle computing device may navigate the vehicle from the initialposition 108 to the first waypoint 112(1). The vehicle computing devicemay also receive a second waypoint 112(2) and a second orientation114(2) and may determine that at least one of the second waypoint 112(2)or the second orientation 114(2) do not pass the safety protocol. Thevehicle computing device may reject the second waypoint 112(2) and/orthe second orientation 114(2) and may cause the vehicle to stop at thefirst waypoint 112(1) to await updated guidance.

In some examples, the vehicle computing system may send a message to theservice computing device indicating the rejection of the waypoint 112and/or the associated orientation 114. In some examples, the message mayinclude a reason for the rejection. For example, the vehicle computingsystem may determine that a waypoint is less than a threshold distancefrom a static object in the environment 100. Based on the determination,the vehicle computing system may reject the waypoint and send a messageto the service computing device indicating that the distance is lessthan the threshold distance.

Responsive to a determination that the waypoint(s) 112 and/or associatedorientation(s) 114 pass the safety protocol, the vehicle computingsystem may accept the waypoint(s) 112 and/or orientation(s) 114. Thevehicle computing system may process the waypoint(s) 112 and/ororientation(s) 114 and generate one or more trajectories 116 for thevehicle 102 to transit between waypoints 112. For example, the vehiclecomputing system may receive a first waypoint 112(1) and a firstorientation 114(1). Based on a determination that the first waypoint112(1) and the first orientation 114(1) are valid, the vehicle computingsystem may determine a first set of trajectories 116 (one or moretrajectories) to transit from an initial position to the first waypoint112(1) at the first orientation 114(1). In at least some examples, theone or more waypoints 112 may be used as constraints in determining sucha trajectory (e.g., the vehicle 102 must pass through the waypoint at aparticularly designated yaw angle). In additional or alternativeexamples, the waypoints may comprise costs in which a trajectory isdetermined based at least in part on a cost associated with how far thevehicle 102 is from a waypoint and/or a difference in the yaw of thevehicle 102 and the corresponding waypoint 112. In any one or more ofthe trajectories determined, a velocity may be constrained not to meetor exceed a threshold velocity, which may be a fixed velocity (e.g., 5mph, 10 mph, etc.) and/or a percentage of the related speed limit overthe drivable surface traversed.

Based on the generated trajectories 116, the vehicle computing systemmay control the vehicle 102 to the waypoint(s) 112 at the associatedorientation(s) 114, such as via one or more drive systems. In variousexamples, the vehicle computing system may receive waypoints 112 and/orassociated orientations 114 incrementally. In such examples, the vehiclecomputing system may control the vehicle from an initial position 108 tothe first waypoint 112(1). In some examples, the vehicle computingsystem may arrive at the first waypoint 112(1) prior to receiving and/orverifying follow-on guidance, such as a second waypoint 112(2). In suchexamples, the vehicle computing system may cause the vehicle to stop atthe first waypoint 112(1) until the second waypoint 112(2) is receivedand validated. In some examples, the vehicle computing system mayreceive and/or verify the second waypoint 112(2) prior to arriving atthe first waypoint 112(1). In such examples, the vehicle computingdevice may control the vehicle to maneuver past the first waypoint112(1) toward the second waypoint 112(2) without stopping. In someexamples, the follow-on guidance may be provided while slowing to a stopat a particular waypoint, such as the first waypoint 112(1). In suchexamples, the vehicle computing device may speed the vehicle back up toan appropriate speed and navigate the vehicle toward the second waypoint112(2).

In various examples, the vehicle computing device may control thevehicle according to a remote guidance protocol. The remote guidanceprotocol may include one or more limitations on vehicle 102 movementwhile operating in the remote guidance mode. The remote guidanceprotocol may include a path deviation (e.g., distance from a pathbetween two waypoints the vehicle 102 may deviate, such as to circumventobstacles, ½ width of the vehicle, 1 vehicle width, etc.), response tostatic object detection (e.g., attempt to circumvent static objects toget to a follow-on waypoint 112, response to dynamic object detection(e.g., yield to agents predicted to enter a path corresponding to thewaypoints 112), follow dynamic objects moving in the same direction asthe vehicle (e.g., in the path corresponding to the waypoints 112),yield as appropriate for right of way at intersections (e.g., four waystops, traffic lights, roundabouts, etc.), stop forward motion if nofurther guidance received (e.g., another waypoint, instruction to resumenavigation, etc.), or the like.

In various examples, the vehicle computing system may control thevehicle 102 at a velocity appropriate for conditions in the environment.The conditions may include a number and/or type of dynamic objectsdetected in the environment (e.g., a number of other vehicles on theroad, a number of pedestrians, children playing in the vicinity, etc.),a zone associated with the environment (e.g., school zone, playground,etc.), or the like. In some examples, the vehicle computing system maybe configured to dynamically determine a maximum velocity of the vehicleto maximize safety while still permitting forward motion. For example,the vehicle computing device may determine that it is in a school zonewhile operating in a guidance mode. The vehicle computing may determinethat a maximum velocity associated with the vehicle 102 is 7 miles perhour, such as to permit an immediate halt of the vehicle 102 upondetection of a student approaching a path associated with the vehicle102. For another example, the vehicle 102 may be following anothervehicle in traffic. The vehicle computing device may control the vehicle102 at a velocity to maintain a safe distance behind the precedingvehicle.

In various examples, the vehicle computing system may control thevehicle 102 below a pre-determined maximum velocity (e.g., 20 kilometersper hour, 15 miles per hour, 10 miles per hour, etc.). In some examples,the pre-determined maximum velocity may include a maximum velocityassociated with remote guidance. In various examples, the vehiclecomputing system may control the vehicle 102 at or below a velocityassociated with the environment and/or the scenario. In at least oneexample, the vehicle computing system may control the vehicle 102 at thelesser of the pre-determined maximum velocity or the maximum velocityassociated with the environment and/or the scenario. For example, avehicle computing system, when operating in a remote guidance mode, mayoperate the vehicle at a pre-determined maximum velocity of 15 miles perhour (e.g., a speed between 0 and 15 miles per hour). The vehiclecomputing system may detect a remote guidance scenario 106 and requestguidance input from a service computing device. The vehicle computingsystem, and/or the operator may determine, based on sensor data, thatthe remote guidance scenario 106 has associated therewith a speed limitof 10 miles per hour. Based on the speed limit of 10 miles per hourbeing less than the pre-determined maximum velocity of 15 miles perhour, the vehicle computing system may limit the velocity of the vehicleto no more than 10 miles per hour while navigating through the remoteguidance scenario 106.

In various examples, the vehicle computing system may be configured toverify that the vehicle 102 will not violate a safety protocol whilenavigating through the remote guidance scenario 106 (e.g., travelingwaypoint to waypoint). The vehicle computing device may continuously,periodically (e.g., every 0.1 second, 0.5 second, 1 second, etc.),and/or at random intervals (e.g., at 3 seconds, 5 seconds, etc.),monitor the surroundings to ensure safety protocols are met by thevehicle (e.g., the vehicle will not impact an object 118 in theenvironment). For example, the vehicle computing device may control thevehicle 102 to ensure that the vehicle 102 maintains a minimum distancefrom an object 118 in the environment.

In various examples, the vehicle computing device may detect one or moreobjects 118 in the environment 100, such as based on sensor data. Theobject(s) 118 may be static and/or dynamic objects. In various examples,the vehicle computing system may be configured to determine aclassification associated with each of the detected object(s) 118 (e.g.,type of object, such as car, semi-trailer truck, pedestrian, bicyclist,etc.). In various examples, the vehicle computing system may determineone or more predicted object trajectories 120 based on the sensor data.The object trajectories 120 may represent any number of potential pathsin which the object 118 may travel through the environment 100 from aposition associated with a time of perception. The object trajectories120 may be determined based on a variety of factors, such as aclassification of the object 118, traffic laws or rules (e.g., rules ofthe road), driving etiquette, a location in a turning/non-turning lane,proximity to an intersection, other stationary and/or dynamic objects,drivable surfaces, etc.

In various examples, the object trajectories 120 may be determinedutilizing one or more machine learning algorithms. In such examples, thevehicle computing system, such as via a prediction system, may receivethe sensor data associated with the object 118 and predict a behavior ofthe object 118 based on the sensor data. For example, the vehiclecomputing system may be trained on captured image data of object motionover time such that the vehicle computing system may recognize behaviorthat may suggest future actions (e.g., object trajectories) the object118 may take. In some examples, the vehicle computing system maydetermine predicted object trajectories 120 based on a top-downrepresentation of an environment, such as by utilizing the techniquesdescribed in U.S. patent application Ser. No. 16/151,607 filed Oct. 4,2018 and entitled “Trajectory Prediction on Top-Down Scenes,” the entirecontents of which are incorporated herein by reference. Additionally, orin the alternative, the vehicle computing system may utilize heat maps,tree search methods, and/or temporal logic formulas, such as thosedescribed in the U.S. patent application Ser. No. 15/807,521 filed Nov.8, 2017 and entitled “Probabilistic Heat Maps for Behavior Prediction,”(the entire contents of which are incorporated herein by reference) todetermine one or more object trajectories 120 associated with thedetected objects 118.

In various examples, the vehicle computing system may process the objecttrajectories 120 associated with detected object(s) 118 in theenvironment 100 and may determine whether the vehicle 102 and theobject(s) 118 may interact (e.g., a collision, near miss, etc.). Forexample, a vehicle computing system may control a vehicle 102 from awaypoint 112(8) to a subsequent waypoint 112(N). The vehicle computingsystem may detect an object 118 (e.g., a pedestrian) with an objecttrajectory 120 that appears to intersect with a trajectory associatedwith the vehicle 102. The vehicle computing device may determine thatthe object 118 may cause the vehicle 102 to violate a safety protocol(e.g., pedestrian within a threshold distance of the vehicle 102 whilein motion, etc.) and/or a remote guidance protocol (e.g., objectcrossing path of the vehicle in remote guidance mode). The vehiclecomputing system may determine an action to take based on thepedestrian, such as yielding to the pedestrian, to ensure the vehicle102 maintains the safety protocol and/or remote guidance protocol.

In various examples, the service computing device may determine that ascenario is complete (e.g., vehicle passes the remote guidance scenario106, can resume normal operations) and may release the vehicle 102 fromthe remote guidance. In some examples, scenario completion may be basedin part on a determination that the vehicle may safely navigate theremainder of the remote guidance scenario 106. For example, an occludedobject that was not visible to the vehicle computing system at theinitial position 108 is detected by the perception system and thevehicle computing system may be configured to circumnavigate theoccluded object. For another example, a first section of the remoteguidance scenario 106 may include a violation of an operations protocol,such as traveling in a lane designated for opposite direction traffic.The second section of the remote guidance scenario 106 may includetraveling within the bounds of the operations protocol. Thus, theoperator may determine that the scenario may be complete based uponcompletion of the first section (e.g., based upon the vehicle transitingin appropriate lanes for the direction of travel).

In some examples, scenario completion may be based in part on agenerated waypoint 112, such as 112(N) being within a threshold distance(e.g., 0.5 meters, 0.5 feet, 10 inches, etc.) of an original routeassociated with the vehicle 102. In some examples, scenario completionmay be based on sensor data input. In such examples, the operator mayview the sensor data via the GUI to determine that the scenario iscomplete. For example, the remote guidance scenario 106 may include aconstruction zone. A camera feed from the vehicle may depict an end ofthe construction zone behind the vehicle 102. Based on a determinationthat the construction zone is behind the vehicle 102, the operator maydetermine that the scenario is complete.

In various examples, responsive to a determination that the scenario iscomplete, the service computing device may send a release signal to thevehicle 102. In such examples, responsive to the release signal, thevehicle computing device may be configured to resume navigation of thevehicle 102. In various examples, the service computing device maydetermine that the scenario blocked a planned route 104 (e.g., initialroute) of the vehicle 102. In such examples, the service computingdevice may be configured to generate an updated route for the vehicle toarrive at a pre-determined destination. In some examples, the servicecomputing device may send the updated route to the vehicle computingdevice prior to sending the release signal. In some examples, theservice computing device may send the updated route substantiallyconcurrently with the release signal (e.g., within 1 second, 2 seconds,etc.). Responsive to receipt of the updated route, the vehicle computingsystem may control the vehicle 102 along the updated route to thepre-determined destination. In some examples, the vehicle computingdevice may determine that the scenario blocked the initial route. Insuch examples, the vehicle computing device may be configured togenerate the updated route to navigate to the destination. In variousexamples, based on a determination that neither the service computingdevice nor the vehicle computing device may have determined an updatedroute, the vehicle computing device may cause the vehicle to pull over(e.g., control the vehicle out of the flow of traffic) and stop todetermine an updated route and/or receive updated route guidance fromthe service computing device.

In various examples, after connecting to the service computing device,but prior to receiving guidance data (e.g., waypoint(s) 112 and/ororientation(s) 114), the vehicle computing system may determine asolution for safely navigating the remote guidance scenario 106. In suchexamples, the vehicle computing system may send a message to the servicecomputing device rescinding the request for remote guidance. In someexamples, the service computing device may launch the GUI to enable theoperator to observe the vehicle 102 navigating autonomously through theremote guidance scenario 106, such as acting as a safety monitor. Insome examples, the service computing device may complete the scenarioand may disable input controls associated with the GUI, denying theoperator from providing waypoint 112 and/or orientation 114 input.

FIG. 2 depicts an example service computing device 200 with a graphicaluser interface 202 (GUI 202) in which an operator 204 (illustrated asinput cursor controlled by operator) may input waypoint 206 and/ororientation 208 guidance, such as waypoints 112 and/or orientations 114,for a vehicle 210, such as vehicle 102, operating in a remote guidancemode. As discussed above, a vehicle computing device associated with thevehicle 210 may detect one or more obstacles 212 comprising a remoteguidance scenario 214, such as remote guidance scenario 106. Responsiveto detecting the remote guidance scenario 214, the vehicle computingsystem may establish a connection with the service computing device 200.In various examples, the vehicle computing system may send a request forguidance data (e.g., guidance input) from the service computing device200. In at least some examples, the vehicle computing system mayrecognize such a remote guidance scenario 214 before having to make afull and complete stop (e.g., can recognize such a scenario far enoughin advance that the techniques discussed herein may cause the vehicle tocontinue in motion to a destination without stopping).

In various examples, responsive to the connection with the vehiclecomputing device and/or receipt of the request for guidance data, theservice computing device 200 may determine whether one or moreconditions for guidance are met. The condition(s) for guidance mayinclude system checks to ensure that the service computing device 200can effectively provide guidance to the vehicle 210. The condition(s)may include a vehicle velocity below a threshold (e.g., less than 15miles per hour, less than 20 kilometers per hour, etc.), a thresholdnumber and/or type of sensors are available (e.g., minimum of fourcameras providing forward, aft, left, and right views operational,sensors associated with aerial perspective view operational, etc.),threshold bandwidth (e.g., equal to or greater than 10 megabytes persecond, 15 megabytes per second, etc.), threshold latency (e.g., equalto or less than 300 milliseconds, 250 milliseconds, etc.), no vehiclehealth related faults (e.g., steering column fault, etc.), or the like.

In some examples, based on a determination that a condition is not met,the service computing device 200 may communicate the lack of a conditionto the vehicle computing device. In such examples, the service computingdevice 200 may be unable to provide guidance data until the conditionsare met (e.g., bandwidth increases, latency decreases, no vehiclehealth-related faults, vehicle speed decreases below the threshold,sensor data availability increases, etc.). In some examples, based on adetermination that the condition is not met, the service computingdevice 200 may not launch the GUI 202. In some examples, based on adetermination that the condition is not met, the service computingdevice 200 may launch the GUI 202, but may not enable input controls. Insuch examples, the operator 204 may be unable to provide waypoint 206and/or orientation 208 guidance. In any such example, the vehicle mayexecute a fallback maneuver (e.g., maintaining a current position and/orpulling over to a safe stop area).

In various examples, based on a determination that the condition(s) aremet, the service computing device 200 may launch the GUI 202 and may notenable input controls. In such examples, the operator 204 may be able toprovide waypoint 206 and/or orientation 208 guidance. The GUI 202 mayinclude one or more windows 216 depicting sensor data from one or moresensors on the vehicle 210. In at least one example, the window(s) 216may depict streaming images captured by a camera on the vehicle 210. Thewindow(s) 216 may depict data to assist the operator in determining apath 218 for the vehicle to take. The path may include a straight linebetween waypoints, which may or may not be indicative of an actual path230 the vehicle may traverse through the environment. In some examples,the difference between the path 218 and an actual path 230 may be based,at least in part, on the orientation 208 or direction the vehicle willface at a waypoint 206, such as the first orientation 208(1) associatedwith the first waypoint 206(1). In such examples, the vehicle computingsystem may generate an actual path 230 indicative of the one or moretrajectories of the vehicle to arrive at the first waypoint 206(1) inthe first orientation 208(1). In such examples, trajectories maycomprise one or more states to obtain (positions, orientations,velocities, accelerations, etc.) over a finite time horizon and/orbetween subsequent waypoints. As above, such waypoints may be used todetermine such trajectories as costs and/or constraints.

In the illustrative example, the GUI 202 includes a first window 216(1)depicting a left front view, a second window 216(2) depicting a frontview, a third window 216(3) depicting a right front view, and a fourthwindow 216(4) depicting a rear view associated with a vehicle 210. Theoperator may assess each of the views to determine a path 218 for thevehicle to transit though the remote guidance scenario 214. The path 218may represent a straight-line distance between an initial position 222of the vehicle 210 and a first waypoint 206(1) and/or a straight-linedistance between waypoints 206, such as between the first waypoint206(1) and a second waypoint 206(2).

In various examples, the GUI 202 may include a depiction of a top view220 (e.g., birds-eye view) of the vehicle 210 and the remote guidancescenario 214 which, in at least some examples, may be adjusted accordingto the operator 204 to provide differing points of view. In someexamples, the top view 220 may include a computer-generated view of theenvironment based on sensor data. In some examples, the top view 220 mayinitially include a position 222 of the vehicle 210 with respect to theobstacle(s) 212 of the remote guidance scenario 214. In the illustrativeexample, the top view 220 depicts the vehicle 210 approaching multipleorange cones blocking a planned path 224 (e.g., initial path, plannedroute, etc.) of the vehicle 210, such as planned route (e.g., plannedpath) 104.

In various examples, the top view 220 may be configured to receiveguidance input from the operator 204. In such examples, the operator 204may be able to provide guidance to the vehicle to navigate through theremote guidance scenario 214, such as around obstacle(s) 212 associatedtherewith. The input may include one or more waypoints 206. Thewaypoint(s) 206 may include locations over which the vehicle willtravel. In various example waypoint(s) 206 may each include stoppingpoints, such as absent further guidance. In such examples, the vehicle210 may be configured to stop at each of the waypoint(s) 206 absentreceipt or verification of a subsequent waypoint 206.

In various examples, the operator 204 may select a waypoint entry option226. In some examples, the waypoint entry option 226 may enable theinput of waypoints 206 into the top view 220. In various examples, theoperator 204 may input a waypoint using an input device associated withthe service computing device 200. Though illustrated as a cursor, suchas associated with a mouse, other input devices are contemplated, suchas touch screens, keyboards, or the like. The operator 204 may move thecursor over the top view 220 to different potential locations for awaypoint 206. In some examples, a symbol associated with a waypoint 206,such as the second waypoint 206(2) may accompany the cursor. In theillustrative example, the second waypoint 206(2) is depicted as having alight color. In such an example, the light color may represent awaypoint 206 that has not yet been entered by the operator 204.Responsive to an entry, such as by a click of a mouse over a particularlocation, the waypoint 206 may be entered and the waypoint 206, such as206(1) may be represented in a different color. In at least someexamples, the GUI 202 may preclude the operator 204 from inputting awaypoint more than a certain distance from a previously entered waypointand/or the last known position of the vehicle.

In various examples, the operator 204 may select an orientation entryoption 228. The orientation entry option may enable the operator 204 toinput an orientation 208 (e.g., yaw) associated with a waypoint 206. Asdiscussed above, the orientation 208 may represent a direction thevehicle should face at a particular waypoint 206. In some examples, theorientation 208 may be determined based on a previous and/or subsequentwaypoint 206. In such examples, the orientation at a particular waypoint206 may enable the vehicle to smoothly transition between waypoints on aguidance route. In various examples, an orientation 208 associated witha waypoint 206 may be based on a preferred orientation 208 should thevehicle be stopped at the waypoint 206.

In some examples, the GUI 202 may be configured to receive orientationentry on waypoints 206 that have been entered (e.g., placementconfirmed). For example, responsive to the operator 204 clicking on(e.g., selecting) the second waypoint 206(2), thereby changing the colorto match entered waypoint 206(1), the operator 204 may be able to inputan orientation 208 associated therewith, such as orientation 208(2). Inthe illustrative example, the operator 204 may move a cursor or otherinput device around the second waypoint 206(2) (e.g., scratch thecursor) and selecting a direction in which the vehicle 210 will face atthe second waypoint 206(2). The direction may represent the orientation208(2) or yaw of the vehicle 210 at the second waypoint 206(2). Asdiscussed above, in various examples, the direction may coincide withthe path 218 generated by the GUI 202. In various examples, the movementof the cursor may be confined by an orientation limitation 242. Theorientation limitation 242 may include a range of directions in whichthe orientation 208 may be set. In various examples, the GUI 202 mayblock orientation input outside the orientation limitation 242. Theorientation limitation 242 may be based on a physical constraint placedon the vehicle 210 (e.g. a maximum yaw rate, maximum yaw angle, maximumlateral distance, maximum turn angle, etc.). For example, the top view220 may not accept orientation input that would cause the vehicle toperform a large turn at a waypoint 206, such as 90-degree or 180-degreeturn.

In various examples, the service computing device 200 may be configuredto validate the waypoint(s) 206 and/or orientation(s) 208 entered viathe GUI. As discussed above, waypoint 206 and/or orientation 208validation may include a verification that the vehicle 210 will notviolate (e.g., pass) a safety protocol and/or a remote guidance protocolassociated with the vehicle 210 navigating according to the waypoint 206and/or orientation 208. In some instances, such validation may comprisea kinematics/dynamics check (validation) which ensures that the vehiclewould be capable of performing such a waypoint transition.

In various examples, responsive to receiving guidance input (e.g.,waypoint(s) 206 and/or orientation(s) 208), the service computing device200 may send the waypoint(s) 206 and/or orientation(s) 208 (e.g.,waypoint data) to the vehicle computing system. In some examples, theservice computing device 200 may send the first waypoint 206(1) and/orfirst orientation 208(1) prior to receiving input associated with thesecond waypoint 206(2) and/or second orientation 208(2). In suchexamples, the vehicle computing system may be configured to receive thefirst waypoint 206(1) and/or first orientation 208(1) and cause thevehicle to start moving from the initial position 222. Absent receipt ofdata associated with the second waypoint 206(2), the vehicle computingsystem may cause the vehicle 210 to stop at the first waypoint 206(1).In various examples, responsive to receipt (and validation) of thesecond waypoint 206(2) and/or second orientation 208(2), the vehiclecomputing system may cause the vehicle to proceed toward the secondwaypoint 206(2). In such examples, the service computing device 200 mayenable incremental remote guidance to the vehicle computing system. Insome examples, the service computing device 200 may send data associatedwith two or more waypoints 206 and/or orientations 208. In suchexamples, based on validation of the two or more waypoints, the vehiclecomputing system may be configured to navigate between the two or morewaypoints 206. In any of the examples described herein, the secondwaypoint 206(2) may be received as the vehicle is traversing to thefirst waypoint 206(1) such that the vehicle may continue through allwaypoints without stopping.

As discussed above, responsive to receiving a waypoint 206 and/or anorientation 208, such as first waypoint 206(1) and/or first orientation208(1), the vehicle computing system may generate one or moretrajectories for the vehicle 210. The trajectories may represent anactual path 230 of the vehicle 210 traveling between locations whileoperating in a guidance mode. In various examples, the trajectoriesassociated with the actual path 230 may include trajectories that willresult in the vehicle arriving at the first waypoint 206(1) in the firstorientation 208(1).

In various examples, the vehicle computing system may send the servicecomputing device an indication that it is unable to proceed beyond acurrent location. The current location may include a waypoint 206, suchas waypoint 206(1), or another location in the environment. In someexamples, the vehicle may stop at the current location based on adetermination that the vehicle computing device cannot generate the oneor more trajectories for the vehicle 210, that a subsequent waypoint206, such as the second waypoint 206(2), has not yet been received, thata path to the second waypoint 206(2) is obstructed, that a waypoint 206was rejected, or the like. In some examples, responsive to receiving theindication, the service computing device may present a notification onthe GUI 202 indicating that the vehicle is unable to proceed. Based onthe notification, the operator may input the second (subsequent)waypoint 206(2) and/or a replacement waypoint 206, such as if the secondwaypoint 206(2) was rejected.

In various examples, the GUI 202 may enable the operator 204 to causethe vehicle to stop at a particular location. In some examples, thevehicle may stop at a waypoint, such as waypoint 206(1), based on adetermination that follow-on guidance has not been received (e.g., dataassociated with second waypoint 206(2) not received and/or verified). Insome examples, the operator 204 may cause the vehicle 210 to stop byselecting the hold option 232. Selection of the hold option 232 maycause the service computing device 200 to send a hold signal to thevehicle 210. In some examples, responsive to receiving the hold signal,the vehicle computing system may control the vehicle to a stop. In someexamples, responsive to receiving the hold signal, the vehicle computingsystem may maintain the vehicle in a stopped position.

In various examples, the vehicle computing system may maintain thestopped position (such as responsive to the hold signal) until acompletion signal (e.g., release signal) is received. The completionsignal may be sent by the service computing device responsive to inputof the release option 234 by the operator 204. In various examples, thecompletion signal may cause the vehicle computing device to control thevehicle based on guidance input received. In some examples, a hold andrelease may enable an operator to send multiple waypoints 206 and/ororientations 208 at one time. In some examples, the waypoints 206 and/ororientations 208 entered during a vehicle hold may be sent substantiallysimultaneously with the completion signal.

In various examples, the GUI 202 may include a completion option 236.The completion option 236 may be selected by the operator responsive toa determination that the vehicle has navigated the remote guidancescenario 214 and may proceed autonomously. In some examples, thecompletion option 236 may be selected responsive to a determination thata terminating condition has occurred. The terminating condition mayinclude a determination that the remote guidance scenario 214 is behindthe vehicle, that the vehicle 210 is within a threshold distance of theplanned path 224, that the vehicle 210 is capable of proceedingautonomously (e.g., controlling the vehicle according to the operationsprotocol, safety protocol, etc.), or the like. In various examples,responsive to selection of the completion option 236, the servicecomputing device 200 may send a completion signal to the vehiclecomputing device. The completion signal may include an instruction toproceed autonomously. In various examples, the service computing device200 may determine that the remote guidance scenario 214 blocked theplanned path 224 of the vehicle 210 and that the vehicle 210 willproceed along another route (e.g., updated route). In some examples theservice computing device 200 may automatically generate an updated routefor the vehicle to navigate to a destination. In some examples, theservice computing device may generate the updated route based on inputby the operator of an update route option 238.

In various examples, the GUI 202 may be configured to receive waypointedits (e.g., deletion or modification), such as responsive to an editwaypoint option 240 of the GUI 202. The edit waypoint option 240 mayenable the operator to delete and/or modify a waypoint 206 (e.g., changea position) and/or an orientation 208 (e.g., change a yaw) associatedtherewith. The waypoints 206 and/or orientations 208 may be edited basedon a determination that a waypoint 206 has not been confirmed (e.g.,accepted by the vehicle computing system) or a determination that thewaypoint has been confirmed but has not been completed by the vehicle(e.g., the vehicle 210 has not arrived at the waypoint 206, the vehiclehas not started along the actual path 230 to the waypoint 206). Invarious examples, responsive to the determination that the waypoint hasbeen confirmed but not completed, the GUI 202 may verify that thevehicle 210 is stopped prior to enabling waypoint 206 editing. In thoseexample in which the waypoint is currently being used by the vehiclecomputing system to control the vehicle, the waypoint may be locked forediting. In such examples, this precludes the operator 204 fromadjusting a waypoint after it has been used for planning (therebycreating a safer planning process).

FIG. 3 depicts an example process 300 for providing incremental guidanceto a vehicle 102. Some or all of the process 300 may be performed by oneor more components described below with regard to FIG. 8.

At operation 302, a service computing device may receive a request forguidance from a vehicle computing system. The vehicle computing systemmay send the request for guidance based on a detection of a remoteguidance scenario 106 in an operating environment. The remote guidancescenario 106 may include an obstacle, road blockage, construction zonein the road, or the like. The remote guidance scenario 106 may include ascenario in which the vehicle computing system is not programmed toperform and/or a scenario that appears to violate an operations protocoland/or a safety protocol associated with the vehicle 102. As above, sucha scenario may be detected in anticipation of the vehicle approachingthe scenario such that guidance can be relayed prior to stopping so thatthe vehicle does not have to stop at all.

Responsive to receiving the request for guidance, the service computingdevice may process the request and launch a graphical user interface(GUI). The GUI may include an interface through which an operator mayprovide guidance input to the vehicle. The GUI may be configured toreceive input corresponding to a first waypoint 112(1) (e.g. firstposition) for the vehicle to navigate the remote guidance scenario 106(e.g., around an obstacle). In some examples, the GUI may be configuredto receive input corresponding to a first orientation associated withthe first waypoint 112(1). The first orientation may represent adirection in which the vehicle 102 may face at the first waypoint112(1).

At operation 304, the service computing device may send the firstwaypoint 112(1) (e.g., data associated with a first position and/or thefirst orientation associated with the first waypoint 112(1)) to thevehicle computing system. The vehicle computing system may receive dataassociated with the first waypoint 112(1) and may control the vehicle tothe first waypoint 112(1). In some examples, the vehicle computingsystem may additionally control the vehicle 102 based on to the firstorientation, such as to cause the vehicle 102 to arrive at the firstwaypoint 112(1) in the first orientation (and/or within some thresholdthereof). In various examples, responsive to not receiving follow-onguidance, the vehicle may stop at the first waypoint 112(1), representedby stop bar 306. In some examples, the vehicle 102 may stop at the stopbar 306 responsive to receiving a hold signal, such as that describedabove with regard to FIG. 2. In such examples, the service computingdevice may send the hold signal responsive to receiving an input fromthe operator to hold the vehicle 102.

At operation 308, the service computing device may send a secondwaypoint 112(2) and a third waypoint 112(3) to the vehicle computingsystem. In various examples, the second waypoint 112(2) and the thirdwaypoint 112(3) may be determined based at least in part on an occludedobject 310 initially detected at the first waypoint 112(1). The occludedobject 310 may include an object that was not initially visible (e.g.,detected by sensor systems) to the vehicle computing system upondetection of the remote guidance scenario 106. In some examples, theservice computing device may be configured to provide incrementalguidance based on a real-time or near real-time detection of obstacles,objects, etc. in the environment.

In various examples, the vehicle computing device may control thevehicle to the second waypoint 112(2) and then to the third waypoints112(3) without stopping at any waypoint 112. In such examples, thevehicle may proceed without stopping based on a determination thatsafety protocols are met (e.g., detected objects will not impact travel,etc.).

At operation 312, the service computing device may determine that ascenario (e.g., remote guidance scenario 106) is complete (e.g.,terminating condition has occurred). In some examples, the servicecomputing device may determine that the scenario is complete based on adetermination, such as using sensor data, that one or more obstaclesassociated with the remote guidance scenario 106 and/or the occludedobject 310 are behind the vehicle. In some examples, the servicecomputing device may determine that the scenario is complete based on adetermination that the third waypoint 112(3) (e.g., a final waypoint) iswithin a threshold distance (e.g., 4 inches, 0.5 feet, 2 feet, 1 meter,etc.) of a planned route 104 of the vehicle. In various examples, thevehicle control system may determine that it is able to plantrajectories for the vehicle 102 that do not violate any control policy(and safely traverse an environment) and send a signal indicative ofsuch to the service computing device. In such examples, the servicecomputing device may determine, based on the signal from the vehiclecomputing system, that the scenario is complete.

At operation 314, the service computing device may release the vehiclefrom remote guidance. In some examples, the service computing device maysend a completion signal to the vehicle computing system. In someexamples, responsive to receiving the completion signal, the vehiclecomputing system may control the vehicle autonomously. In some examples,the completion signal may include an instruction to proceedautonomously. In such examples, responsive to receiving the instruction,the vehicle computing system may control the vehicle autonomously.

FIG. 4 depicts an operator workflow 400 for providing remote guidance toa vehicle via a graphical user interface 402. At operation 404, avehicle computing system may connect to a service computing device. Theservice computing device may determine that one or more condition(s) forproviding remote guidance is not satisfied. In some examples, theservice computing device may determine that the condition(s) are notsatisfied based on data received from the vehicle computing systemand/or data associated with a network connection. The condition(s) forguidance may include system checks to ensure that the service computingdevice can effectively provide guidance to the vehicle. The condition(s)may include a current vehicle velocity below a threshold (e.g., lessthan 15 miles per hour, less than 20 kilometers per hour, etc.), athreshold number and/or type of sensors are available (e.g., minimum offour cameras providing forward, aft, left, and right views operational,sensors associated with aerial perspective view operational, etc.),threshold bandwidth (e.g., equal to or greater than 10 megabytes persecond, 15 megabytes per second, etc.), threshold latency (e.g., equalto or less than 300 milliseconds, 250 milliseconds, etc.), no vehiclehealth related faults (e.g., steering column fault, etc.), or the like.

Based on a determination that at least one condition of the one or moreconditions is not satisfied, the service computing device may determine,at operation 406, that guidance is not possible. In various examples,based on a determination that the at least one condition of thecondition(s) is not satisfied, the service computing device may, atoperation 408, block guidance from being input into the graphical userinterface 402. In such an example, an operator may be unable to inputwaypoint and/or orientation data into the graphical user interface 402.

At operation 410, the vehicle computing system may connect to theservice computing device and determine that the condition(s) forproviding remote guidance is/are satisfied. Based on a determinationthat the condition(s) is/are satisfied, the service computing device maydetermine, at operation 412, that guidance is possible. Responsive to adetermination that a condition fails while providing guidance, theservice computing device may determine that guidance is not possible,such as that depicted at operation 406. In some examples, based on adetermination that guidance is not possible, the vehicle may not proceedbeyond an initial position (e.g., may stay stopped). In some examples,based on a determination that guidance is not possible, the servicecomputing device may send the vehicle computing system a messageindicating that remote guidance is not possible. In such examples, themessage may include a justification for not providing remote guidance(e.g., reason that guidance is not possible). Conversely, responsive toa determination that the at least one condition that previously failed,such as that determined at operation 404, is currently satisfied, theservice computing device may determine that guidance is possible(operation 412).

Based on a determination that guidance is possible, the workflow mayinclude, at operation 414, activating guidance on the graphical userinterface. In various examples, activating guidance may enable theoperator to input data corresponding to waypoints and/or orientations toassist the vehicle in navigating around an obstacle. The operator maythus build a path for the vehicle around the obstacle.

Based on a determination that at least one condition of the condition(s)fails (same or a different condition from the condition described asfailing above) while the operator builds the path, further guidance maybe blocked at operation 408. In some examples, based on a determinationthat the at least one condition fails, the vehicle may not proceedbeyond an initial position (e.g., may stay stopped) and/or a mostrecently received and/or validated waypoint. In such examples, thevehicle computing system may be configured to cause the vehicle to stopresponsive to not receiving further guidance (e.g., subsequentwaypoints, completion signal, etc.). Further, at operation 408,responsive to a determination that the at least one condition is met,the guidance may be re-activated on the graphical user interface 402,enabling additional waypoint and/or orientation data input.

FIG. 5 is a flow diagram 500 depicting vehicle actions responsive toremote guidance input. At operation 502, an operator associated with agraphical user interface of a service computing device may connect to avehicle computing system. In various examples, the operator may connectto the vehicle computing system responsive to receiving a request forremote guidance. In some examples, the connection may be based on asatisfaction of one or more conditions.

At operation 504, the vehicle may be stopped in hold. In variousexamples, the vehicle may be stopped, such as by the vehicle computingdevice, based on a detection of a remote guidance scenario. In someexamples, the vehicle may be stopped based on an instruction to hold,such as that received from a service computing device via a graphicaluser interface. The vehicle may be stopped in a Go (e.g., movementauthorized) or a NoGo (e.g., movement not authorized) mode. The Go modemay represent a mode in which a temporary delay is necessary prior toforward progress. In some examples, the temporary delay may be based ona time necessary to input and/or validate a waypoint. In some examples,the temporary delay may be based on a satisfaction of a safety protocoland/or remote guidance protocol. For example, the vehicle computingdevice may detect another vehicle in front of the vehicle has stopped.To maintain a safe distance behind the other vehicle, the vehiclecomputing device may stop the vehicle in Go mode. The NoGo mode mayrepresent a mode in which remote guidance may not be available, such asbased on a condition not being met (e.g., a detected vehicle healthfault, etc.), or otherwise that the vehicle has performed a failsafemaneuver and has diverged from nominal driving behavior.

At operation 506, the vehicle may be moving in release. In someexamples, the vehicle may move in release based on receipt of a releasesignal from the service computing device associated with the graphicaluser interface. In various examples, the vehicle may request remoteguidance while moving. From operation 504 or operation 506, an operatormay start a guidance mode 508 on a graphical user interface of theservice computing device.

At operation 510, the vehicle may be stopped in hold. In some examples,the guidance mode 508 may be active while the vehicle is holding. Insuch examples, the operator may input waypoints and/or orientations forguidance around an obstacle (e.g., navigating a remote guidancescenario); however, due to the hold, the vehicle may not control thevehicle to traverse the environment according to the waypoints.

At operation 512, the vehicle may continue to move operating normalplanner guidance (e.g., autonomous control provided by the vehiclecomputing system) until data associated with a waypoint is received fromthe service computing device. Responsive to the operator adding awaypoint, the vehicle, at operation 514, may satisfy one or morewaypoints received (e.g., drive to each waypoint). In some examples,responsive to receiving a waypoint, the vehicle computing system mayvalidate the waypoint to ensure the waypoint satisfies a safety protocoland/or a remote guidance protocol of the vehicle.

In various examples, responsive to receiving a waypoint, the vehicle maygenerate one or more trajectories to control the vehicle to thewaypoint. At operation 514, the vehicle computing device may receivedata corresponding to one or more additional waypoints from the servicecomputing device (e.g., operator adding additional waypoints in thegraphical user interface). The vehicle computing system may continuallygenerate trajectories and control the vehicle between subsequentwaypoints based on the trajectories.

In various examples, a vehicle stopped in hold at operation 510 may bereleased, such as by the operator selecting a release option on thegraphical user interface, or the like. In such examples, the vehiclemay, at operation 514, start moving and satisfying the remainingwaypoints in the path.

At operation 516, the vehicle may stop and await additional guidance. Insome examples, the vehicle may stop at operation 516 based on adetermination that it has not yet received follow-on guidance (asubsequent waypoint). In such examples, responsive to receiving awaypoint, the vehicle may start moving again (operation 514). In someexamples, the vehicle may stop based on a determination that the finalwaypoint has been satisfied (e.g., vehicle arrived at final waypoint).In some examples, responsive to completion of the guidance, the operatormay complete guidance and disconnect from the vehicle computing system,such as at operation 520. In some examples, the operator may cause theservice computing device to send a completion message (e.g., completionsignal) to the vehicle computing system indicating that the guidancepath is complete. Responsive to guidance path completion, the vehiclecomputing system may resume controlling the vehicle autonomously.

In various examples, the vehicle may continue moving, such as atoperation 514 until, at operation 518, the vehicle satisfies a finalwaypoint in the path. In some examples, the operator may indicateguidance completion prior to the final waypoint, such as by designatinga waypoint as the final waypoint. In some examples, responsive tocompletion of the guidance, the operator may complete guidance anddisconnect from the vehicle computing system, such as at operation 520.Responsive to guidance path completion, the vehicle computing system mayresume controlling the vehicle autonomously. In various examples, theservice computing device may receive a second request for remoteguidance. In such examples, the operator may re-connect to the vehiclecomputing system at operation 502.

FIGS. 6 and 7 illustrate example processes in accordance withembodiments of the disclosure. These processes are illustrated aslogical flow graphs, each operation of which represents a sequence ofoperations that may be implemented in hardware, software, or acombination thereof. In the context of software, the operationsrepresent computer-executable instructions stored on one or morecomputer-readable storage media that, when executed by one or moreprocessors, perform the recited operations. Generally,computer-executable instructions include routines, programs, objects,components, data structures, and the like that perform particularfunctions or implement particular abstract data types. The order inwhich the operations are described is not intended to be construed as alimitation, and any number of the described operations may be combinedin any order and/or in parallel to implement the processes.

FIG. 6 depicts an example process 600 for providing remote guidance to avehicle computing device. Some or all of the process 600 may beperformed by one or more components in FIG. 8, as described herein. Forexample, some or all of the process 600 may be performed by the servicecomputing device(s) 832.

At operation 602, the process may include receiving a request forguidance input for navigating a scenario from a vehicle computing systemassociated with a vehicle. The request for guidance input may bereceived via a network connection between the vehicle computing systemand the service computing device. The request for guidance input mayinclude a request for incremental guidance through the scenario, such asvia one or more waypoints and/or associated orientations.

At operation 604, the process may include determining whether one ormore conditions are satisfied. The conditions may include a vehiclevelocity below a threshold, a threshold number and/or type of sensorsare available, a threshold bandwidth associated with the networkconnection, a threshold latency associated with the network connection,absence of a vehicle health related fault, or the like.

Responsive to a determination that at least one condition is notsatisfied (“No” at operation 604), the process may include, at operation606, disabling an input capability of a graphical user interface (GUI)of the service computing device. In some examples, responsive to thedetermination that the at least one condition is not satisfied, theservice computing device may not launch the GUI.

Responsive to a determination that the conditions are satisfied (“Yes”at operation 604), the process may include, at operation 608, receivinginput corresponding to a waypoint. The input may be received from anoperator associated with the GUI, such as via an input/output deviceassociated with the service computing device. In various examples, theinput may include a position and/or orientation (e.g., yaw) of awaypoint. In such examples, the position and/or orientation of thewaypoint may represent a location the vehicle will pass while operatingin a guidance mode, and a direction the vehicle will face at thelocation.

In various examples, the service computing device may validate thewaypoint and/or orientation associated therewith. A validation of thewaypoint may include a verification that the waypoint and/or orientationassociated therewith satisfies a safety protocol and/or a remoteguidance protocol associated with vehicle operations. The safetyprotocol may include one or more rules associated with the safeoperation of the vehicle, such as a location of the waypoint being on aroad (e.g., not less than ½ width of the vehicle from the edge of theroad or drivable surface), the waypoint being within a thresholddistance of an initial position or within a threshold distance ofanother waypoint, one or more threshold distances for the vehicle tomaintain away from dynamic and/or static objects, maximum yaw rates,and/or other criteria associated with safe navigation of the vehiclethrough an environment. In additional, or alternative examples, suchvalidation may comprise kinematics and/or dynamics checks to ensure thevehicle is capable of performing transitions from one waypoint toanother.

At operation 610, the process may include sending data associated withthe waypoint to the vehicle computing system. In some examples, the datamay be sent to the vehicle computing system based on a validation of theassociated waypoint and/or orientation. In some examples, the data mayinclude a position and/or orientation associated with one or morewaypoints. In various examples, the vehicle computing system mayvalidate the waypoint(s) and control the vehicle based in part on thewaypoint(s).

At operation 612, the process may include determining whether thescenario navigation is complete. In various examples, the servicecomputing device may determine that the scenario is complete based on atransmission of a final waypoint associated with the scenario. In someexamples, the service computing device may identify the final waypointbased on the waypoint being at least threshold distance (e.g., 5 feet,10 feet, 3 meters, etc.) past an obstacle associated with the scenario.In some examples, the service computing device may identify the finalwaypoint based on the waypoint being less than a threshold distance(e.g., 4 inches, 1 foot, 1 meter, etc.) from an initial route associatedwith the vehicle.

Responsive to a determination that the scenario navigation is notcomplete (“No” at operation 612), the service computing device mayreturn to operation 608 and receive input corresponding to a waypoint.In various examples, the service computing device may continue toreceive input corresponding to a waypoint, send data associated with thewaypoint to the vehicle computing system until the scenario navigationis complete.

Responsive to a determination that the scenario navigation is complete(“Yes” at operation 612), the process may include, at operation 614,sending a completion signal to the vehicle computing system. In someexamples, the completion signal may include an instruction to proceedautonomously to the destination. Responsive to receiving the completionsignal, the vehicle computing system may resume normal planner componentfunctionality to autonomously control the vehicle.

In various examples, the service computing device may determine that aninitial route associated with the vehicle was blocked due to thescenario. In such examples, the service computing device may determinethat the vehicle may not be able to follow the initial route to adestination. In some examples, the service computing device may generatean updated route. In such examples, the service computing device maysend the updated route to the vehicle computing device, such asconcurrently with or within a threshold time (e.g., 0.5 seconds, 2seconds, etc.) of sending the completion signal.

FIG. 7 depicts an example process 700 for controlling a vehicle based onguidance input received from a service computing device. Some or all ofthe process 700 may be performed by one or more components in FIG. 8, asdescribed herein. For example, some or all of the process 700 may beperformed by the vehicle computing device(s) 804 (e.g., vehiclecomputing system).

At operation 702, the process may include receiving sensor data from asensor of the vehicle. The sensors may include cameras, motiondetectors, lidar, radar, time of flight, or the like. The vehiclecomputing device may be configured, such as by using a perceptioncomponent, to detect and/or classify one or more dynamic and/or staticobjects in an environment in which the vehicle operates.

In various examples, the vehicle computing system may process the sensordata to determine one or more actions of the vehicle to take whileoperating in the environment. The action(s) may be determined based onthe dynamic and/or static object(s) detected in the environment. Theaction(s) may be based on an operations protocol and/or a safetyprotocol associated with the vehicle. The operations protocol mayinclude one or more rules governing actions the vehicle may or may nottake such as not crossing a double yellow line, not driving intooncoming traffic lanes, not exceeding the bounds of a drivable surfaceof a road, or the like. The safety protocol may include one or morerules associated with the safe operation of the vehicle, one or morethreshold distances for the vehicle to maintain away from detecteddynamic and/or static objects, maximum yaw rates, and/or other criteriaassociated with safe navigation of the vehicle through the environment.In some examples, the vehicle computing device may process the sensordata and may be unable to determine an action to take.

At operation 704, the process may include determining, based on thesensor data, whether the vehicle has encountered a remote guidancescenario. The remote guidance scenario may include an obstacle, roadblockage, construction zone in the road, driveline beyond vehicleline-of-sight (e.g., sensor vision impaired due to elevation of roadand/or turns in a road), or the like. The remote guidance scenario mayinclude a scenario in which the vehicle computing system is notprogrammed to perform and/or a scenario that appears to violate theoperations protocol.

Responsive to determining that the vehicle has not encountered a remoteguidance scenario (“No” at operation 704), the process may include, atoperation 706, the vehicle computing system autonomously controlling thevehicle, such as in a normal planner mode.

Responsive to determining that the vehicle has encountered a remoteguidance scenario (“Yes” at operation 704), the process may include, atoperation 708, sending a request for guidance to a service computingdevice. In various examples, the vehicle computing system may establisha network connection with the service computing device. In suchexamples, the vehicle computing system may send the request via thenetwork connection. In some examples, an observer in the vehicle maymanually establish the network connection between the vehicle computingsystem and the service computing device. In some examples, the vehiclecomputing system may automatically establish the network connectionand/or send the request for guidance responsive to determining that thevehicle has encountered the remote guidance scenario.

At operation 710, the process may include receiving data associated witha waypoint. The data may include a position and/or an orientationassociated with the waypoint.

At operation 712, the process may include determining whether thewaypoint is valid. A validation of the waypoint may include averification that the waypoint and/or orientation associated therewithsatisfies a safety protocol, a remote guidance protocol, and/or akinematics validation associated with vehicle operations. The safetyprotocol may include one or more rules associated with the safeoperation of the vehicle, such as a location of the waypoint being on aroad (e.g., not less than ½ width of the vehicle from the edge of theroad or drivable surface), the waypoint being within a thresholddistance of an initial position or within a threshold distance ofanother waypoint, one or more threshold distances for the vehicle tomaintain away from dynamic and/or static objects, maximum yaw rates,and/or other criteria associated with safe navigation of the vehiclethrough an environment. The remote guidance protocol may include one ormore limitations on vehicle movement while operating in the remoteguidance mode. The remote guidance protocol may include a path deviation(e.g., distance from a path between two waypoints the vehicle maydeviate, such as to circumvent obstacles, ½ width of the vehicle, 1vehicle width, etc.), response to static object detection (e.g., attemptto circumvent static objects to get to a follow-on waypoint, response todynamic object detection (e.g., yield to agents predicted to enter apath corresponding to the waypoints), follow dynamic objects moving inthe same direction as the vehicle (e.g., in the path corresponding tothe waypoints), yield as appropriate for right of way at intersections(e.g., four way stops, traffic lights, roundabouts, etc.), stop forwardmotion if no further guidance received (e.g., another waypoint,instruction to resume navigation, etc.), or the like. The kinematicsvalidation may include a verification that the vehicle computing systemmay generate one or more trajectories to navigate the vehicle to thewaypoint at the associated orientation.

Responsive to determining that the waypoint and/or orientation isinvalid (“No” at operation 712), the process may include, at operation714, rejecting the waypoint. In various examples, the vehicle computingsystem may send a message to the service computing device indicating therejection of the waypoint. In various examples, the message may includeone or more reasons for determining that the waypoint is invalid (e.g.,explanation of rejection).

Responsive to determining that the vehicle has encountered a remoteguidance scenario (“Yes” at operation 712), the process may include, atoperation 716, controlling the vehicle based at least in part on thedata associated with the waypoint (e.g., controlling the vehicle at theinclusion of the waypoint). As a non-limiting example, such a waypoint(and/or associated orientation) may be used as one or more of a cost orcontrol for planning a trajectory from one waypoint to the next. Controlof the vehicle may include sending one or more signals to one or moredrive systems associated with the vehicle and causing the vehicle toeither hold a position or move to a different position, such as thatassociated with the waypoint, and face a particular direction, such asthat associated with a determined orientation corresponding to thewaypoint.

In at least some examples, if the scenario is not surpassed, such aprocess may continue to 710 whereby additional waypoints are receivedand considered. Otherwise, process 700 may continue to 706 to continueautonomously controlling the vehicle.

FIG. 8 is a block diagram of an example system 800 for implementing thetechniques described herein. In at least one example, the system 800 mayinclude a vehicle 802, such as vehicle 102.

The vehicle 802 may include one or more vehicle computing devices 804,one or more sensor systems 806, one or more emitters 808, one or morecommunication connections 810, at least one direct connection 812, andone or more drive systems 814.

The vehicle computing device(s) 804 may include one or more processors816 and memory 818 communicatively coupled with the one or moreprocessors 816. In the illustrated example, the vehicle 802 is anautonomous vehicle; however, the vehicle 802 could be any other type ofvehicle, such as a semi-autonomous vehicle, or any other system havingat least an image capture device (e.g., a camera enabled smartphone). Inthe illustrated example, the memory 818 of the vehicle computingdevice(s) 804 stores a localization component 820, a perceptioncomponent 822, a planning component 824, one or more system controllers826, and one or more maps 828. Though depicted in FIG. 8 as residing inthe memory 818 for illustrative purposes, it is contemplated that thelocalization component 820, a perception component 822, a planningcomponent 824, one or more system controllers 826, and one or more maps828 may additionally, or alternatively, be accessible to the vehicle 802(e.g., stored on, or otherwise accessible by, memory remote from thevehicle 802, such as, for example, on memory 834 of a service computingdevice(s) 832).

In at least one example, the localization component 820 may includefunctionality to receive data from the sensor system(s) 806 to determinea position and/or orientation of the vehicle 802 (e.g., one or more ofan x-, y-, z-position, roll, pitch, or yaw). For example, thelocalization component 820 may include and/or request/receive a map ofan environment, such as from map(s) 828 and may continuously determine alocation and/or orientation of the autonomous vehicle within the map. Invarious examples, the map(s) 828 may include road segmentsidentifications (e.g., numbers, etc.) associated with various portionsof the roads on the map(s). In some examples, the road segmentidentifications may be associated with a road network managed at leastin part by the service computing device(s) 832. In some instances, thelocalization component 820 may utilize SLAM (simultaneous localizationand mapping), CLAMS (calibration, localization and mapping,simultaneously), relative SLAM, bundle adjustment, non-linear leastsquares optimization, or the like to receive image data, lidar data,radar data, IMU data, GPS data, wheel encoder data, and the like toaccurately determine a location of the autonomous vehicle. In someinstances, the localization component 820 may provide data to variouscomponents of the vehicle 802 to determine an initial position of anautonomous vehicle for determining whether an object is relevant to thevehicle 802, as discussed herein.

In some instances, the perception component 822 may includefunctionality to perform object detection, segmentation, and/orclassification. In some examples, the perception component 822 mayprovide processed sensor data that indicates a presence of an object(e.g., entity) that is proximate to the vehicle 802 and/or aclassification of the object as an object type (e.g., car, pedestrian,cyclist, animal, building, tree, road surface, curb, sidewalk, unknown,etc.). In some examples, the perception component 822 may provideprocessed sensor data that indicates a presence of a stationary entitythat is proximate to the vehicle 802 and/or a classification of thestationary entity as a type (e.g., building, tree, road surface, curb,sidewalk, unknown, etc.). In additional or alternative examples, theperception component 822 may provide processed sensor data thatindicates one or more characteristics associated with a detected object(e.g., a tracked object) and/or the environment in which the object ispositioned. In some examples, characteristics associated with an objectmay include, but are not limited to, an x-position (global and/or localposition), a y-position (global and/or local position), a z-position(global and/or local position, e.g., height), an orientation (e.g., aroll, pitch, yaw), an object type (e.g., a classification), a velocityof the object, an acceleration of the object, an extent of the object(size), etc. Characteristics associated with the environment mayinclude, but are not limited to, a presence of another object in theenvironment, a state of another object in the environment, a zoneassociated with the environment (e.g., school zone, business district,etc.), a time of day, a day of a week, a season, a weather condition, anindication of darkness/light, etc.

In general, the planning component 824 may determine a path, such asplanned route 104 (e.g., initial route, planned path, etc.), for thevehicle 802 to follow to traverse through an environment. For example,the planning component 624 may determine various routes and trajectoriesand various levels of detail. For example, the planning component 824may determine a route to travel from a first location (e.g., a currentlocation) to a second location (e.g., a target location). In someexamples, the planning component 824 may generate an instruction forguiding the autonomous vehicle 802 along at least a portion of the routefrom a first location to the second location.

In various examples, the planning component 824 may be configured toreceive data associated with waypoints and/or orientations for thevehicle 802 operating in a remote guidance mode from a remote guidanceplatform 830 of one or more service computing devices 832. In someexamples, the planning component 824 may be configured to validate thewaypoints and/or orientations. In at least one example, the planningcomponent 824 may determine how to guide the autonomous vehicle 802 froma first waypoint received from the service computing device(s) 832 to asecond waypoint received from the service computing device(s) 832. Insome examples, the instruction may be a trajectory, or a portion of atrajectory. In some examples, multiple trajectories may be substantiallysimultaneously generated (e.g., within technical tolerances) inaccordance with a receding horizon technique, wherein one of themultiple trajectories is selected for the vehicle 802 to navigate.

In some examples, the planning component 824 may include a predictioncomponent to generate predicted trajectories of objects (e.g., objects)in an environment. For example, a prediction component may generate oneor more predicted trajectories for dynamic objects detected in anenvironment. In some examples, the predicted trajectories may includeany number of potential paths in which a detected object may travel froma current position (e.g., at the time of perception) and/or based on adirection of travel. In some examples, the predicted trajectories mayrepresent a distance and direction in which the object will travel overa period of time. In some examples, a prediction component may measure atrace of an object and generate a trajectory for the object based onobserved and predicted behavior. In various examples, the predictioncomponent may determine predicted trajectories based on one or more ofmachine learning techniques, heat maps, temporal logic, and/or treesearch methods.

In at least one example, the vehicle computing device(s) 804 may includeone or more system controllers 826, which may be configured to controlsteering, propulsion, braking, safety, emitters, communication, andother systems of the vehicle 802. The system controller(s) 826 maycommunicate with and/or control corresponding systems of the drivesystem(s) 814 and/or other components of the vehicle 802.

The memory 818 may further include one or more maps 828 that may be usedby the vehicle 802 to navigate within the environment. For the purposeof this discussion, a map may be any number of data structures modeledin two dimensions, three dimensions, or N-dimensions that are capable ofproviding information about an environment, such as, but not limited to,topologies (such as intersections), streets, mountain ranges, roads,terrain, and the environment in general. In some instances, a map mayinclude, but is not limited to: texture information (e.g., colorinformation (e.g., RGB color information, Lab color information, HSV/HSLcolor information), and the like), intensity information (e.g., lidarinformation, radar information, and the like); spatial information(e.g., image data projected onto a mesh, individual “surfels” (e.g.,polygons associated with individual color and/or intensity)),reflectivity information (e.g., specularity information,retroreflectivity information, BRDF information, BSSRDF information, andthe like). In one example, a map may include a three-dimensional mesh ofthe environment. In some examples, the vehicle 802 may be controlledbased at least in part on the map(s) 828. That is, the map(s) 828 may beused in connection with the localization component 820, the perceptioncomponent 822, and/or the planning component 824 to determine a locationof the vehicle 802, detect objects in an environment, generate routes,determine actions and/or trajectories to navigate within an environment.In various examples, the map(s) 828 may include a road network. The roadnetwork may include one or more distinct road segments.

In some examples, the one or more maps 828 may be stored on one or moreservice computing device(s) 832. In some examples, multiple maps 828 maybe stored based on, for example, a characteristic (e.g., type of entity,time of day, day of week, season of the year, etc.). Storing multiplemaps 828 may have similar memory requirements but increase the speed atwhich data in a map may be accessed.

As can be understood, the components discussed herein (e.g., thelocalization component 820, the perception component 822, the planningcomponent 824, the one or more system controller(s) 826, and the one ormore maps 828 are described as divided for illustrative purposes.However, the operations performed by the various components may becombined or performed in any other component.

In some instances, aspects of some or all of the components discussedherein may include any models, techniques, and/or machine learningtechniques. For example, in some instances, the components in the memory818 (and the memory 834, discussed below) may be implemented as a neuralnetwork.

As described herein, an exemplary neural network is a biologicallyinspired technique which passes input data through a series of connectedlayers to produce an output. Each layer in a neural network may alsocomprise another neural network, or may comprise any number of layers(whether convolutional or not). As can be understood in the context ofthis disclosure, a neural network may utilize machine learning, whichmay refer to a broad class of such techniques in which an output isgenerated based on learned parameters.

Although discussed in the context of neural networks, any type ofmachine learning may be used consistent with this disclosure. Forexample, machine learning techniques may include, but are not limitedto, regression techniques (e.g., ordinary least squares regression(OLSR), linear regression, logistic regression, stepwise regression,multivariate adaptive regression splines (MARS), locally estimatedscatterplot smoothing (LOESS)), instance-based techniques (e.g., ridgeregression, least absolute shrinkage and selection operator (LASSO),elastic net, least-angle regression (LARS)), decisions tree techniques(e.g., classification and regression tree (CART), iterative dichotomiser3 (ID3), Chi-squared automatic interaction detection (CHAD), decisionstump, conditional decision trees), Bayesian techniques (e.g., naïveBayes, Gaussian naïve Bayes, multinomial naïve Bayes, averageone-dependence estimators (AODE), Bayesian belief network (BNN),Bayesian networks), clustering techniques (e.g., k-means, k-medians,expectation maximization (EM), hierarchical clustering), associationrule learning techniques (e.g., perceptron, back-propagation, hopfieldnetwork, Radial Basis Function Network (RBFN)), deep learning techniques(e.g., Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN),Convolutional Neural Network (CNN), Stacked Auto-Encoders),Dimensionality Reduction Techniques (e.g., Principal Component Analysis(PCA), Principal Component Regression (PCR), Partial Least SquaresRegression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS),Projection Pursuit, Linear Discriminant Analysis (LDA), MixtureDiscriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA),Flexible Discriminant Analysis (FDA)), Ensemble Techniques (e.g.,Boosting, Bootstrapped Aggregation (Bagging), AdaBoost, StackedGeneralization (blending), Gradient Boosting Machines (GBM), GradientBoosted Regression Trees (GBRT), Random Forest), SVM (support vectormachine), supervised learning, unsupervised learning, semi-supervisedlearning, etc. Additional examples of architectures include neuralnetworks such as ResNet50, ResNet101, VGG, DenseNet, PointNet, and thelike.

In at least one example, the sensor system(s) 806 may include lidarsensors, radar sensors, ultrasonic transducers, sonar sensors, locationsensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertialmeasurement units (IMUs), accelerometers, magnetometers, gyroscopes,etc.), cameras (e.g., RGB, IR, intensity, depth, time of flight, etc.),microphones, wheel encoders, environment sensors (e.g., temperaturesensors, humidity sensors, light sensors, pressure sensors, etc.), etc.The sensor system(s) 806 may include multiple instances of each of theseor other types of sensors. For instance, the lidar sensors may includeindividual lidar sensors located at the corners, front, back, sides,and/or top of the vehicle 802. As another example, the camera sensorsmay include multiple cameras disposed at various locations about theexterior and/or interior of the vehicle 802. The sensor system(s) 806may provide input to the vehicle computing device(s) 804.

The vehicle 802 may also include one or more emitters 808 for emittinglight and/or sound. The emitter(s) 808 may include interior audio andvisual emitters to communicate with passengers of the vehicle 802. Byway of example and not limitation, interior emitters may includespeakers, lights, signs, display screens, touch screens, haptic emitters(e.g., vibration and/or force feedback), mechanical actuators (e.g.,seatbelt tensioners, seat positioners, headrest positioners, etc.), andthe like. The emitter(s) 808 may also include exterior emitters. By wayof example and not limitation, the exterior emitters may include lightsto signal a direction of travel or other indicator of vehicle action(e.g., indicator lights, signs, light arrays, etc.), and one or moreaudio emitters (e.g., speakers, speaker arrays, horns, etc.) to audiblycommunicate with pedestrians or other nearby vehicles, one or more ofwhich comprising acoustic beam steering technology.

The vehicle 802 and/or the service computing device(s) 832 may alsoinclude one or more communication connection(s) 810 that enablecommunication between the vehicle 802 and the service computingdevice(s) 832 and/or other local computing device(s) on the vehicle 802and/or the drive system(s) 814. Also, the communication connection(s)810 may allow the vehicle 802 to communicate with other nearby computingdevice(s) (e.g., other nearby vehicles, etc.).

The communications connection(s) 810 may include physical and/or logicalinterfaces for connecting the vehicle computing device(s) 804 to anothercomputing device or a network, such as network(s) 836. For example, thecommunications connection(s) 810 can enable Wi-Fi-based communicationsuch as via frequencies defined by the IEEE 802.11 standards, shortrange wireless frequencies such as Bluetooth, cellular communication(e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.) or any suitable wired or wirelesscommunications protocol that enables the respective computing device tointerface with the other computing device(s).

In at least one example, the vehicle 802 may include one or more drivesystems 814. In some examples, the vehicle 802 may have a single drivesystem 814. In at least one example, if the vehicle 802 has multipledrive systems 814, individual drive system(s) 814 may be positioned onopposite ends of the vehicle 802 (e.g., the front and the rear, etc.).In at least one example, the drive system(s) 814 may include one or moresensor systems to detect conditions of the drive system(s) 814 and/orthe surroundings of the vehicle 802. By way of example and notlimitation, the sensor system(s) may include one or more wheel encoders(e.g., rotary encoders) to sense rotation of the wheels of the drivesystems, inertial sensors (e.g., inertial measurement units,accelerometers, gyroscopes, magnetometers, etc.) to measure orientationand acceleration of the drive system, cameras or other image sensors,ultrasonic sensors to acoustically detect objects in the surroundings ofthe drive system, lidar sensors, radar sensors, etc. Some sensors, suchas the wheel encoders may be unique to the drive system(s) 814. In somecases, the sensor system(s) on the drive system(s) 814 may overlap orsupplement corresponding systems of the vehicle 802 (e.g., sensorsystem(s) 806).

The drive system(s) 814 may include many of the vehicle systems,including a high voltage battery, a motor to propel the vehicle, aninverter to convert direct current from the battery into alternatingcurrent for use by other vehicle systems, a steering system including asteering motor and steering rack (which can be electric), a brakingsystem including hydraulic or electric actuators, a suspension systemincluding hydraulic and/or pneumatic components, a stability controlsystem for distributing brake forces to mitigate loss of traction andmaintain control, an HVAC system, lighting (e.g., lighting such ashead/tail lights to illuminate an exterior surrounding of the vehicle),and one or more other systems (e.g., cooling system, safety systems,onboard charging system, other electrical components such as a DC/DCconverter, a high voltage junction, a high voltage cable, chargingsystem, charge port, etc.). Additionally, the drive system(s) 814 mayinclude a drive system controller which may receive and preprocess datafrom the sensor system(s) and to control operation of the variousvehicle systems. In some examples, the drive system controller mayinclude one or more processors and memory communicatively coupled withthe one or more processors. The memory may store one or more systems toperform various functionalities of the drive system(s) 814. Furthermore,the drive system(s) 814 may also include one or more communicationconnection(s) that enable communication by the respective drive systemwith one or more other local or remote computing device(s).

In at least one example, the direct connection(s) 812 may provide aphysical interface to couple the one or more drive system(s) 814 withthe body of the vehicle 802. For example, the direct connection(s) 812may allow the transfer of energy, fluids, air, data, etc. between thedrive system(s) 814 and the vehicle. In some instances, the directconnection(s) 812 may further releasably secure the drive system(s) 814to the body of the vehicle 802.

In at least one example, the localization component 820, the perceptioncomponent 822, the planning component 824, the one or more systemcontrollers 826, and the one or more maps 828 may process sensor data,as described above, and may send their respective outputs, over the oneor more networks 836, to the service computing device(s) 832. In atleast one example, the localization component 820, the perceptioncomponent 822, the planning component 824, the one or more systemcontrollers 826, and the one or more maps 828 may send their respectiveoutputs to the service computing device(s) 832 at a particularfrequency, after a lapse of a predetermined period of time, in nearreal-time, etc.

In some examples, the vehicle 802 may send sensor data to the servicecomputing device(s) 832 via the network(s) 836. In some examples, thevehicle 802 may receive sensor data from the service computing device(s)832 via the network(s) 636. The sensor data may include raw sensor dataand/or processed sensor data and/or representations of sensor data. Insome examples, the sensor data (raw or processed) may be sent and/orreceived as one or more log files.

The service computing device(s) 832 may include one or more processors842 and the memory 834 storing the remote guidance component 830 with agraphical user interface 838. The graphical user interface 838 may beconfigured to receive input from an operator representative of one ormore waypoints and/or orientations for providing remote guidance to thevehicle 802, such as that described above. The input may be received viaone or more input/output devices 840. The input/output device(s) 840 mayinclude one or more of a keyboard, a mouse, a touch screen display, ahaptic device, microphone, camera, and/or any other device configured toinput and/or output data to/from the service computing device(s) 832.

The processor(s) 816 of the vehicle 802 and the processor(s) 842 of theservice computing device(s) 832 may be any suitable processor capable ofexecuting instructions to process data and perform operations asdescribed herein. By way of example and not limitation, the processor(s)816 and 842 may comprise one or more Central Processing Units (CPUs),Graphics Processing Units (GPUs), or any other device or portion of adevice that processes electronic data to transform that electronic datainto other electronic data that may be stored in registers and/ormemory. In some examples, integrated circuits (e.g., ASICs, etc.), gatearrays (e.g., FPGAs, etc.), and other hardware devices may also beconsidered processors in so far as they are configured to implementencoded instructions.

Memory 818 and 834 are examples of non-transitory computer-readablemedia. The memory 818 and 834 may store an operating system and one ormore software applications, instructions, programs, and/or data toimplement the methods described herein and the functions attributed tothe various systems. In various implementations, the memory may beimplemented using any suitable memory technology, such as static randomaccess memory (SRAM), synchronous dynamic RAM (SDRAM),nonvolatile/Flash-type memory, or any other type of memory capable ofstoring information. The architectures, systems, and individual elementsdescribed herein may include many other logical, programmatic, andphysical components, of which those shown in the accompanying figuresare merely examples that are related to the discussion herein.

In some instances, the memory 818 and 834 may include at least a workingmemory and a storage memory. For example, the working memory may be ahigh-speed memory of limited capacity (e.g., cache memory) that is usedfor storing data to be operated on by the processor(s) 816 and 842. Insome instances, the memory 818 and 834 may include a storage memory thatmay be a lower-speed memory of relatively large capacity that is usedfor long-term storage of data. In some cases, the processor(s) 816 and842 cannot operate directly on data that is stored in the storagememory, and data may need to be loaded into a working memory forperforming operations based on the data, as discussed herein.

Example Clauses

A: A system comprising: one or more processors; and memory storingprocessor-executable instructions that, when executed by the one or moreprocessors, configure the system to: receive, from a vehicle computingsystem associated with a vehicle operating on a route, a request forguidance to navigate around an obstacle in an environment; receivesensor data associated with one or more sensors of the vehicle;determine that a condition for providing remote guidance is met;generate a signal configured to cause a display to output arepresentation of the vehicle and the obstacle on a user interfaceassociated with the guidance system, the representation being based atleast in part on the sensor data; receive, via the user interface, afirst input comprising a first waypoint associated with a first positionand a first orientation, wherein the first orientation comprises a firstdirection for the vehicle to face at the first waypoint; transmit thefirst waypoint to the vehicle computing system; receive, via the userinterface, a second input comprising a second waypoint associated with asecond position and a second orientation, wherein the second orientationcomprises a second direction for the vehicle to face at the secondwaypoint; transmit the second waypoint to the vehicle computing system;determine that a terminating condition for providing guidance is met;and transmit an instruction to the vehicle computing system to controlthe vehicle autonomously.

B: A system as paragraph A describes, wherein the condition comprises atleast one of: a latency corresponding to a connection with the vehiclecomputing system being below a threshold latency; a bandwidthcorresponding to the connection with the vehicle computing system beingbelow a threshold bandwidth; a velocity of the vehicle being below athreshold velocity; a number of sensors available to provide the sensordata is above a threshold number; or a determination that the vehicledoes not comprise an associated health related fault.

C: A system as either one of paragraphs A or B describe, wherein theinstructions further configure the system to: determine that the firstposition, the first orientation, the second position and the secondorientation satisfy at least one of a safety protocol associated withoperation of the vehicle or a remote guidance protocol, the remoteguidance protocol comprising one or more limitations associated withvehicle movement while operating in a remote guidance mode; and validatethe first position, the first orientation, the second position, and thesecond orientation based at least in part on satisfying the at least oneof the safety protocol or the remote guidance protocol.

D: A system as any one of paragraphs A-C describe, wherein determiningthat the terminating condition for providing guidance is met comprises:determining that the second waypoint is within a threshold distance ofthe route; and determining, based at least in part on the thresholddistance, that the obstacle is behind the vehicle.

E: A system as any one of paragraphs A-D describe, wherein theinstructions configure the system to: determine that the vehicle isunable to proceed along the route to a destination; generate an updatedroute for vehicle guidance to the destination; and send the updatedroute to the vehicle computing system, wherein the instruction toproceed autonomously includes the updated route.

F: A method comprising: receiving, from a vehicle computing deviceassociated with a vehicle, a request for guidance input for navigatingaround an obstacle; receiving, as received data and from the vehiclecomputing device, sensor data or data based at least in part on thesensor data; receiving, via a user interface associated with a guidancesystem, a first input corresponding to a first waypoint, the first inputbeing based at least in part on the received data; sending first dataassociated with the first waypoint to the vehicle computing device; andcausing the vehicle to be controlled based at least in part on the firstwaypoint.

G: A method as paragraph F describes, further comprising determiningthat a condition for providing remote guidance is satisfied, thecondition comprising at least one of: a latency corresponding to aconnection with the vehicle computing device being below a thresholdlatency; a bandwidth corresponding to the connection with the vehiclecomputing device being below a threshold bandwidth; a velocity of thevehicle being below a threshold velocity; a number of sensors availableto provide the received data is equal to or greater than a thresholdnumber; or a determination that the vehicle does not comprise anassociated health related fault.

H: A method as either of paragraphs F or G describe, further comprising:determining, at a first time, that a condition for providing remoteguidance is satisfied; enabling inputs into the user interface based atleast in part on the condition being satisfied; determining, at a secondtime, that the condition for providing remote guidance is not satisfied;and disabling the inputs into the user interface based at least in parton the condition not being satisfied.

I: A method as any one of paragraphs F-H describe, further comprising:determining that the first waypoint satisfies at least one of a safetyprotocol associated with operation of the vehicle or a remote guidanceprotocol, the remote guidance protocol comprising one or morelimitations associated with vehicle movement while operating in a remoteguidance mode; and validating the first waypoint based at least in parton satisfying the at least one of the safety protocol or the remoteguidance protocol, wherein sending the first data is based at least inpart on validating the first waypoint.

J: A method as any one of paragraphs F-I describe, further comprising:receiving a second input corresponding to a second waypoint; sendingsecond data associated with the second waypoint to the vehicle computingdevice; and receiving a rejection message from the vehicle indicatingrejection of the second waypoint based at least in part on at least oneof a safety protocol or a remote guidance protocol corresponding tooperation of the vehicle.

K: A method as any one of paragraphs F-J describe, wherein the firstinput comprises: a first location associated with the first waypoint,the first location corresponding to a selectable area of the userinterface; and a first orientation corresponding to a first directionthe vehicle will face at the first waypoint.

L: A method as any one of paragraphs F-K describe, further comprising:sending a hold signal to the vehicle, the hold signal causing thevehicle computing device to hold at a position; and sending a releasesignal to the vehicle, wherein the release signal comprises aninstruction to the vehicle computing device to control the vehicle tothe first waypoint.

M: A method as any one of paragraphs F-L describe, receiving a secondinput corresponding to a second waypoint; sending second data associatedwith the second waypoint to the vehicle computing device; determiningthat the second waypoint corresponds to a terminating conditionassociated with the obstacle; and sending an instruction to the vehiclecomputing device to autonomously control the vehicle after completing anoperation associated with the second waypoint.

N: A method as any one of paragraphs F-M describe, further comprising:receiving a second input corresponding to a second waypoint; sendingsecond data associated with the second waypoint to the vehicle computingdevice; receiving, via the user interface, a third input indicating anoperator intention to edit at least one of the first waypoint or thesecond waypoint; determining that the vehicle is in transit to the firstwaypoint; disabling a first edit to the first waypoint; determining thatat least one of the second waypoint has not been accepted by the vehiclecomputing device or the vehicle has not advanced beyond the firstwaypoint; enabling a second edit to the second waypoint; and sending aninstruction to the vehicle computing device to at least one of delete ormodify the second waypoint.

O: A system or device comprising: a processor; and a non-transitorycomputer-readable medium storing instructions that, when executed, causea processor to perform a method as any one of paragraphs F-N describe.

P: A system or device comprising: a means for processing; and a meansfor storing coupled to the means for processing, the means for storingincluding instructions to configure one or more devices to perform amethod as any one of paragraphs F-N describe.

Q: A non-transitory computer-readable medium storing instructions that,when executed, cause one or more processors to perform operationscomprising: receiving, from a vehicle computing device associated with avehicle, a request for guidance input for navigating around an obstacle;receiving, as received data and from the vehicle computing device,sensor data or data based at least in part on the sensor data;receiving, via a user interface associated with a guidance system, afirst input corresponding to a first waypoint, the first input beingbased at least in part on the received data; sending first dataassociated with the first waypoint to the vehicle computing device; andcausing the vehicle to be controlled based at least in part on the firstwaypoint.

R: A non-transitory computer-readable medium as paragraph Q describes,the operations further comprising: receiving, via the user interface, asecond input indicating a command for the vehicle to stop; and sendingan instruction to the vehicle computing device to stop at a locationassociated with a current location.

S: A non-transitory computer-readable medium as either of paragraphs Qor R describe, the operations further comprising: determining that thevehicle is stopped at a location; and sending an instruction to releasethe vehicle from a stopped position, wherein the vehicle may proceedforward based at least in part on the instruction.

T: A non-transitory computer-readable medium as any one of paragraphsQ-S describe, the operations further comprising: receiving a secondinput corresponding to a second waypoint; sending second data associatedwith the second waypoint to the vehicle computing device; receiving, viathe user interface, a third input indicating an operator intention toedit at least one of the first waypoint or the second waypoint;determining that the vehicle is in transit to the first waypoint;disabling a first edit to the first waypoint; determining that at leastone of the second waypoint has not been accepted by the vehiclecomputing device or the vehicle has not advanced beyond the firstwaypoint; enabling a second edit to the second waypoint; and sending aninstruction to the vehicle computing device to at least one of delete ormodify the second waypoint.

U: A non-transitory computer-readable medium as any one of paragraphsQ-T describe, the operations further comprising: receiving an indicationthat the vehicle is unable to proceed from a location; presenting, viathe user interface, the indication that the vehicle is unable to proceedfrom the location; receiving, via the user interface, a second inputcorresponding to a second waypoint; and transmitting the second waypointto the vehicle computing system.

V: A non-transitory computer-readable medium as any one of paragraphsQ-U describe, the operations further comprising: determining that thefirst waypoint satisfies a kinematics check associated with vehiclecontrol; and validating the first waypoint based at least in part onsatisfying the kinematics check, wherein causing the vehicle to becontrolled is based at least in part on validating the first waypoint.

W: A vehicle comprising: a sensor; one or more processors; and memorystoring processor-executable instructions that, when executed by the oneor more processors, configure the vehicle to: determine a path for thevehicle to traverse through an environment to reach a destination;receive sensor data from the sensor; generate, based at least in part onthe sensor data, a first trajectory for the vehicle to traverse alongthe path; determine, based at least in part on the first trajectory,that the vehicle is unable to continue along the path based at least inpart on one or more of an obstacle or a control policy; transmit, to aservice computing device, a request for guidance data; receive, from theservice computing device, waypoint data comprising a waypoint associatedwith a position and an orientation; validate the waypoint based at leastin part on a safety protocol; based at least in part on a validation ofthe waypoint, determine a second trajectory for the vehicle to navigatethe vehicle from an initial position and an initial vehicle orientationto the position and the orientation associated with the waypoint; andcontrol the vehicle based at least in part on the second trajectory.

X: A vehicle as paragraph W describes, wherein the instructions furtherprogram the one or more processors to: receive a message indicatingrelease of the service computing device as a remote guidance platform;determine a third trajectory to control the vehicle from the waypointalong the path; and control the vehicle based at least in part on thethird trajectory.

Y: A vehicle as either one of paragraphs W or X describe, wherein thewaypoint data comprises a first waypoint data comprising a firstwaypoint associated with a first position and a first orientation andwherein the instructions further configure the vehicle to: receive, fromthe service computing device, second waypoint data comprising a secondwaypoint associated with a second position and a second orientation;validate the second waypoint based at least in part on the safetyprotocol; and based at least in part on validating the second waypoint,determine a third trajectory for the vehicle to navigate the vehiclefrom the first waypoint to the second waypoint; and control the vehiclefrom the first waypoint to the second waypoint based at least in part onthe third trajectory, wherein the vehicle is controlled from the firstwaypoint to the second waypoint without stopping forward motion.

Z: A vehicle as any one of paragraphs W-Z describe, wherein theinstructions further configure the vehicle to: detect, based at least inpart on the sensor data, a dynamic object in the environment; determinean object trajectory associated with the object; determine that theobject the object trajectory will traverse within a threshold distanceof the vehicle traveling on the trajectory for the vehicle; and controlthe vehicle based at least in part on the object.

AA: A vehicle as any one of paragraphs A-D describe, wherein theinstructions further configure the vehicle to: receive second waypointdata associated with a second position and a second vehicle orientation;determine that at least one of the second waypoint or the second vehicleorientation violates the safety protocol; and cause the vehicle to stopat the first waypoint.

AB: A method comprising: receiving sensor data from a sensor of avehicle operating along a path in an environment; determining that thevehicle is unable to continue along the path based at least in part on acontrol policy; sending, to a service computing device, a request forguidance data, the guidance data comprising a waypoint to facilitatecontrol of the vehicle through the scenario; receiving waypoint dataassociated with the waypoint from the service computing device;determining that the waypoint is a valid waypoint; and controlling thevehicle based at least in part on the waypoint.

AC: A method as paragraph AB describes, further comprising: receiving avehicle orientation associated with the waypoint; determining that thevehicle orientation is a valid vehicle orientation; and controlling thevehicle based at least in part on the vehicle orientation.

AD: A method as either one of paragraphs AB or AC describes, whereindetermining that the waypoint is the valid waypoint comprisesdetermining at least one of: determining that the waypoint is located ona drivable surface; determining that an operation of the vehiclenavigating to the waypoint does not violate a remote guidance protocol,the remote guidance protocol comprising a limitation associated withvehicle movement while operating in a remote guidance mode; ordetermining that the operation of the vehicle navigating to the waypointdoes not violate a safety protocol, wherein the safety protocolcomprises a condition to ensure the safety of the vehicle.

AE: A method as any one of paragraphs AB-AD describe, wherein thescenario is identified in an environment associated with the vehicle,the method further comprising: receiving, from the service computingdevice, an updated route for navigating through the environment based atleast in part on the scenario; receiving a message indicating release ofthe service computing device as a remote guidance platform; andcontrolling the vehicle based at least in part on the updated route.

AF: A method as any one of paragraphs AB-AE describe, wherein thescenario is identified on a route associated with the vehicle, themethod further comprising: receiving, from the service computing device,a message indicating release of the service computing device as a remoteguidance platform; controlling the vehicle based at least in part on theroute; identifying a second scenario on the route that violates theoperating constraint; sending, to the service computing device, a secondrequest for guidance data; determining a solution to the second scenarioindependent of input from the service computing device; and controllingthe vehicle based at least in part on the solution.

AG: A method as any one of paragraphs AB-AF describe, wherein waypointdata associated with the waypoint comprises first data associated with afirst waypoint, the method further comprising: receiving second waypointdata associated with a second position and a second vehicle orientation;determining that at least one of the second waypoint or thecorresponding vehicle orientation violates a safety protocol; and basedat least in part on determining that the second waypoint violates thesafety protocol, causing the vehicle to stop at the first waypoint.

AH: A method as any one of paragraphs AB-AG describe, whereindetermining that the vehicle is unable to continue along the pathcomprises: determining the path for the vehicle to traverse through anenvironment; generating, based at least in part on the sensor data, atrajectory for the vehicle to traverse along the path; and determinethat the trajectory violates the control policy.

AI: A method as any one of paragraphs AB-AH describe, furthercomprising: receiving second waypoint data associated with a secondwaypoint and a corresponding vehicle orientation; validating the secondwaypoint and the corresponding vehicle orientation; and controlling thevehicle from the first waypoint to the second waypoint without stoppingforward motion.

AJ: A method as any one of paragraphs AB-AI describe, wherein thecontrol policy comprises at least one of: a traffic law or rule; a ruleof good driving; or an obstacle in a path of the vehicle.

AK: A system or device comprising: a processor; and a non-transitorycomputer-readable medium storing instructions that, when executed, causea processor to perform a method as any one of paragraphs AB-AJ describe.

AL: A system or device comprising: a means for processing; and a meansfor storing coupled to the means for processing, the means for storingincluding instructions to configure one or more devices to perform amethod as any one of paragraphs AB-AJ describe.

AM: A non-transitory computer-readable medium storing instructions that,when executed, cause one or more processors to perform operationscomprising: receiving sensor data from a sensor; determining that thevehicle is unable to continue along a path based at least in part on acontrol policy; sending, to a service computing device, a request forguidance data, the guidance data comprising a waypoint to facilitatecontrol of the vehicle through the scenario; receiving waypoint dataassociated with the waypoint from the service computing device;determining that the waypoint is a valid waypoint; and controlling thevehicle based at least in part on the waypoint.

AN: A non-transitory computer-readable medium as paragraph AM describes,the operations further comprising: receiving orientation datacorresponding to an orientation associated with the waypoint;determining that the orientation is a valid orientation; and controllingthe vehicle based at least in part on the orientation.

AO: A non-transitory computer-readable medium as either one ofparagraphs AM or AN describe, wherein controlling the vehicle based atleast in part on the waypoint comprises: determining a vehicletrajectory from an initial position associated with identifying thescenario to the waypoint; and causing a drive system to operate thevehicle based at least in part on the vehicle trajectory.

AP: A non-transitory computer-readable medium as any one of paragraphsAM-AO describe, wherein the waypoint data associated with the waypointcomprises first waypoint data associated with a first waypoint receivedat a first time, the operations further comprising: receiving, from theservice computing device, second waypoint data associated with a secondwaypoint at a second time; determining that the second waypoint isvalid; and controlling the vehicle based at least in part on the secondwaypoint, wherein the vehicle is configured to transition from the firstwaypoint to the second waypoint without stopping.

AQ: A non-transitory computer-readable medium as any one of paragraphsAM-AP describe, wherein the waypoint data associated with the waypointcomprises first waypoint data associated with a first waypoint receivedat a first time, the operations further comprising: determining that thevehicle is within a threshold distance of the first waypoint;determining that second waypoint data associated with a second waypointhas not been received; and causing the vehicle to stop at the firstwaypoint.

AR: A non-transitory computer-readable medium as any one of paragraphsAM-AQ describe, wherein the scenario is identified in an environmentassociated with the vehicle, the operations further comprising:receiving, from the service computing device, a message indicatingrelease of the service computing device as a remote guidance platform;controlling the vehicle based at least in part on the route; identifyinga second scenario on the route that violates the operating constraint;sending, to the service computing device, a second request for guidancedata; determining a solution to the second scenario independent of inputfrom the service computing device; and controlling the vehicle based atleast in part on the solution.

While the example clauses described above are described with respect toone particular implementation, it should be understood that, in thecontext of this document, the content of the example clauses may also beimplemented via a method, device, system, a computer-readable medium,and/or another implementation. Additionally, any of examples A-AR may beimplemented alone or in combination with any other one or more of theexamples A-AR.

CONCLUSION

While one or more examples of the techniques described herein have beendescribed, various alterations, additions, permutations and equivalentsthereof are included within the scope of the techniques describedherein.

In the description of examples, reference is made to the accompanyingdrawings that form a part hereof, which show by way of illustrationspecific examples of the claimed subject matter. It is to be understoodthat other examples can be used and that changes or alterations, such asstructural changes, can be made. Such examples, changes or alterationsare not necessarily departures from the scope with respect to theintended claimed subject matter. While the steps herein may be presentedin a certain order, in some cases the ordering may be changed so thatcertain inputs are provided at different times or in a different orderwithout changing the function of the systems and methods described. Thedisclosed procedures could also be executed in different orders.Additionally, various computations that are herein need not be performedin the order disclosed, and other examples using alternative orderingsof the computations could be readily implemented. In addition to beingreordered, the computations could also be decomposed intosub-computations with the same results.

What is claimed is:
 1. A vehicle comprising: a sensor; one or moreprocessors; and memory storing processor-executable instructions that,when executed by the one or more processors, configure the vehicle to:determine a path for the vehicle to traverse through an environment toreach a destination; receive sensor data from the sensor; generate,based at least in part on the sensor data, a first trajectory for thevehicle to traverse along the path; determine, based at least in part onthe first trajectory, that the vehicle is unable to continue along thepath based at least in part on one or more of an obstacle or a controlpolicy; transmit, to a service computing device, a request for guidancedata; receive, from the service computing device, waypoint datacomprising a waypoint associated with a position and an orientation;validate the waypoint based at least in part on a safety protocol; basedat least in part on a validation of the waypoint, determine a secondtrajectory for the vehicle to navigate the vehicle from an initialposition and an initial vehicle orientation to the position and theorientation associated with the waypoint; and control the vehicle basedat least in part on the second trajectory.
 2. The vehicle as claim 1recites, wherein the instructions further program the one or moreprocessors to: receive a message indicating release of the servicecomputing device as a remote guidance platform; determine a thirdtrajectory to control the vehicle from the waypoint along the path; andcontrol the vehicle based at least in part on the third trajectory. 3.The vehicle as claim 1 recites, wherein the waypoint data comprises afirst waypoint data comprising a first waypoint associated with a firstposition and a first orientation and wherein the instructions furtherconfigure the vehicle to: receive, from the service computing device,second waypoint data comprising a second waypoint associated with asecond position and a second orientation; validate the second waypointbased at least in part on the safety protocol; and based at least inpart on validating the second waypoint, determine a third trajectory forthe vehicle to navigate the vehicle from the first waypoint to thesecond waypoint; and control the vehicle from the first waypoint to thesecond waypoint based at least in part on the third trajectory, whereinthe vehicle is controlled from the first waypoint to the second waypointwithout stopping forward motion.
 4. The vehicle as claim 1 recites,wherein the instructions further configure the vehicle to: detect, basedat least in part on the sensor data, a dynamic object in theenvironment; determine an object trajectory associated with the object;determine that the object the object trajectory will traverse within athreshold distance of the vehicle traveling on the trajectory for thevehicle; and control the vehicle based at least in part on the object.5. The vehicle as claim 1 recites, wherein the instructions furtherconfigure the vehicle to: receive second waypoint data associated with asecond position and a second vehicle orientation; determine that atleast one of the second waypoint or the second vehicle orientationviolates the safety protocol; and cause the vehicle to stop at the firstwaypoint.
 6. A method comprising: receiving sensor data from a sensor ofa vehicle operating along a path in an environment; determining that thevehicle is unable to continue along the path based at least in part on acontrol policy; sending, to a service computing device, a request forguidance data, the guidance data comprising a waypoint to facilitatecontrol of the vehicle through the scenario; receiving waypoint dataassociated with the waypoint from the service computing device;determining that the waypoint is a valid waypoint; and controlling thevehicle based at least in part on the waypoint.
 7. The method as claim 6recites, further comprising: receiving a vehicle orientation associatedwith the waypoint; determining that the vehicle orientation is a validvehicle orientation; and controlling the vehicle based at least in parton the vehicle orientation.
 8. The method as claim 6 recites, whereindetermining that the waypoint is the valid waypoint comprisesdetermining at least one of: determining that the waypoint is located ona drivable surface; determining that an operation of the vehiclenavigating to the waypoint does not violate a remote guidance protocol,the remote guidance protocol comprising a limitation associated withvehicle movement while operating in a remote guidance mode; ordetermining that the operation of the vehicle navigating to the waypointdoes not violate a safety protocol, wherein the safety protocolcomprises a condition to ensure the safety of the vehicle.
 9. The methodas claim 6 recites, wherein the scenario is identified in an environmentassociated with the vehicle, the method further comprising: receiving,from the service computing device, an updated route for navigatingthrough the environment based at least in part on the scenario;receiving a message indicating release of the service computing deviceas a remote guidance platform; and controlling the vehicle based atleast in part on the updated route.
 10. The method as claim 6 recites,wherein the scenario is identified on a route associated with thevehicle, the method further comprising: receiving, from the servicecomputing device, a message indicating release of the service computingdevice as a remote guidance platform; controlling the vehicle based atleast in part on the route; identifying a second scenario on the routethat violates the operating constraint; sending, to the servicecomputing device, a second request for guidance data; determining asolution to the second scenario independent of input from the servicecomputing device; and controlling the vehicle based at least in part onthe solution.
 11. The method as claim 6 recites, wherein waypoint dataassociated with the waypoint comprises first data associated with afirst waypoint, the method further comprising: receiving second waypointdata associated with a second position and a second vehicle orientation;determining that at least one of the second waypoint or thecorresponding vehicle orientation violates a safety protocol; and basedat least in part on determining that the second waypoint violates thesafety protocol, causing the vehicle to stop at the first waypoint. 12.The method as claim 6 recites, wherein determining that the vehicle isunable to continue along the path comprises: determining the path forthe vehicle to traverse through an environment; generating, based atleast in part on the sensor data, a trajectory for the vehicle totraverse along the path; and determine that the trajectory violates thecontrol policy.
 13. The method as claim 6 recites, further comprising:receiving second waypoint data associated with a second waypoint and acorresponding vehicle orientation; validating the second waypoint andthe corresponding vehicle orientation; and controlling the vehicle fromthe first waypoint to the second waypoint without stopping forwardmotion.
 14. The method as claim 6 recites, wherein the control policycomprises at least one of: a traffic law or rule; a rule of gooddriving; or an obstacle in a path of the vehicle.
 15. A non-transitorycomputer-readable medium storing instructions that, when executed, causeone or more processors to perform operations comprising: receivingsensor data from a sensor; determining that the vehicle is unable tocontinue along a path based at least in part on a control policy;sending, to a service computing device, a request for guidance data, theguidance data comprising a waypoint to facilitate control of the vehiclethrough the scenario; receiving waypoint data associated with thewaypoint from the service computing device; determining that thewaypoint is a valid waypoint; and controlling the vehicle based at leastin part on the waypoint.
 16. The non-transitory computer-readable mediumof claim 15, the operations further comprising: receiving orientationdata corresponding to an orientation associated with the waypoint;determining that the orientation is a valid orientation; and controllingthe vehicle based at least in part on the orientation.
 17. Thenon-transitory computer-readable medium of claim 15, wherein controllingthe vehicle based at least in part on the waypoint comprises:determining a vehicle trajectory from an initial position associatedwith identifying the scenario to the waypoint; and causing a drivesystem to operate the vehicle based at least in part on the vehicletrajectory.
 18. The non-transitory computer-readable medium of claim 15,wherein the waypoint data associated with the waypoint comprises firstwaypoint data associated with a first waypoint received at a first time,the operations further comprising: receiving, from the service computingdevice, second waypoint data associated with a second waypoint at asecond time; determining that the second waypoint is valid; andcontrolling the vehicle based at least in part on the second waypoint,wherein the vehicle is configured to transition from the first waypointto the second waypoint without stopping.
 19. The non-transitorycomputer-readable medium of claim 15, wherein the waypoint dataassociated with the waypoint comprises first waypoint data associatedwith a first waypoint received at a first time, the operations furthercomprising: determining that the vehicle is within a threshold distanceof the first waypoint; determining that second waypoint data associatedwith a second waypoint has not been received; and causing the vehicle tostop at the first waypoint.
 20. The non-transitory computer-readablemedium of claim 15, wherein the scenario is identified in an environmentassociated with the vehicle, the operations further comprising:receiving, from the service computing device, a message indicatingrelease of the service computing device as a remote guidance platform;controlling the vehicle based at least in part on the route; identifyinga second scenario on the route that violates the operating constraint;sending, to the service computing device, a second request for guidancedata; determining a solution to the second scenario independent of inputfrom the service computing device; and controlling the vehicle based atleast in part on the solution.